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SUMMARY 


Objective 

Tlie  major  purpose  of  this  stidy  was  to  develop  and  evaluate  a computer-based  problem-oriented 
medical  curriculum  (POMC)  In  the  Physician  Assistant  (PA)  course  at  Sheppard  AFB,  Texas.  Specifically, 
the  objectives  were  to;  (a)  research  and  develop  a problem-oriented  medical  curriculum  in  a portion  of  the 
PA  course,  (b)  conduct  comparative  evaluation  of  the  instructional  effectiveness  of  the  POMC  with  the 
traditional  lecture  curriculum,  and  (c)  document  development  procedures  and  lessons  learned. 

Background 

The  instructional  approach  employed  in  the  conventional  lecture  version  of  the  PA  course  conasted 
of  a linear  sequencing  of  isolated  instructional  material  which  often  lacked  integration  with,  and  practice  in, 
clinical  diagnosis.  In  contrast,  the  new  POMC  was  envisioned  as  a means  of  integrating  medical  material 
while  providing  functional  practice  of  diagnostic  skills  in  the  context  ot  computer-simulated  patient 
encounters.  In  brief,  the  POMC  was  to  insta'ct  students  in  an  essential  body  of  medical  knowledge  in  a 
clinical  context.  Tire  POMC  included  functional  reaming  and  applied  practice  in  solving  clinical  problems  by 
requiring  students  to;  (a)  gather  subjective  data  (history),  (b)  gather  objective  data  (physical  and  lab  exam), 
(c)  analyze  diagnostic  data,  (d)  formulate  plans  of  action,  (e)  prescribe  treatments,  and  (f)  obtain  feedback 
on  treatment  effectiveness. 

Instructional  development  and  delivery  of  the  POMC  was  accomplished  by  means  of  16  inieractive 
computer  tcmiinals  (PLATO  IV)  with  graphic  capability  wliich  afforded  a high  degree  of  individualized 
instruction  through  tutorial  methods.  Approximately  10  percent  of  the  total  instructional  hours  during  the 
first  year  of  training  was  administered  to  School  of  Healtlv  Care  Sciences  students  enrolled  in  the  PA  course 
at  Sheppard  AFB,  Texas. 

Specifics 

Many  of  the  experimental  findings  were  inconclusive  due  to  implementation  constraints  which 
precluded  rigorous  comparisons.  However,  initial  application  of  computer-assisted  instmetion  (CAfi  in 
medical  training  proved  useful  as  lessons  learned  for  conduct  of  subsequent  research  currently  underway  at 
Sheppard  AFB,  Texas.  For  example,  within  lesson  development,  author-coded  divisions  of  labor  were  more 
effective  when  the  coder  was  competent  in  the  subject  matter,  and  the  author' and  coder  were  physically 
located  at  the  same  station  for  ease  of  communication.  Situations  in  which  these  necessary  conditions  were 
absent  proved  ineffective  and  resulted  in  the  fusion  of  the  author  coder  responsibilities  within  the  author 
role.  Other  lessons  learned  contributing  to  the  conduct  of  the  follow-on  effort  included;  (a)  provisions  for 
ensuring  identical  criterion  measures  adrrdnistered  at  the  lesson  test  and  block  test  levels,  (b)  high  student 
flow  courses,  (c)  faculty  involvement  in  the  selection,  development,  and  review  of  instruction  mediated  by 
computer  terminals,  (d)  pre-course  measurement  of  learner  characteristics  to  assist  in  formative  and 
summative  evaluation,  and  (e)  separate  CAl  and  control  groups  for  an  entire  block  of  instruction. 

The  follow-on  investigation  is  cunently  in  progress  with  improved  evaluation  and  implementation 
procedures  designed  to  ascertain  the  comparative  instructional  effectiveness,  time  savings,  and  delivery  co^^ 
differences  between  CAl  and  conventional  instruction  (lectures  and  programmed  instructional  text).  Three 
different  courses  representing  widespread  differences  in  medical  tasks  and  student  aptitudes  have  been 
selected  for  investigation.  The  follow-on  effort  benefited  greatly  from  the  experiences  and  numerous 
lessons  learned  in  the  PA  initial  application  of  CAl  in  medical  training.  Discussion  of  lessons  learned  and 
recommendations  are  incorporated  within  the  mam  body  of  tliis  report. 

Conclusions 

The  present  report  provides  preliminary  information  on  development  procedures,  implementation 
conditions,  student  reactions,  lessons  learned,  and  cost  data  on  the  POMC/CAI  effort.  Subsequent  in  depth 
study  of  the  comparative  effectiveness  of  CAl  as  opposed  to  alternative  modes  (programmed  instructional 
text  and  lecture)  of  instruction  is  currently  in  the  formative  evaluation  stage  in  three  medical  courses  at 
Sheppard  >VFB,  Texas.  Redirection  of  the  effort  to  liigh  student  flow  courses  more  suitable  to  comparative 
analyses  between  CAl  and  non-CAl  modes  of  instruction  commenced  in  March  1976.  This  follow-on  effort 
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win  provide  dedsion-makers  with  data  on  the  comparative  instructional  effectiveness,  time  savings 
differences,  and  delivery  costs  of  CAI  and  non-CAl  modes  of  instruction. 
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COMPUTER  ASSISTED  INSTRUCTION  IN  AIR  FORCE  MEDICAL  TRAINING: 
PRELIMINARY  FINDINGS 


L INTRODUCTION 

Providing  suiUcient  numben  of  competent  health  care  personnel  in  the  least  time  possible  at  an 
acceptable  cost  to  meet  Air  Force  needs  poses  a decision-rnaking  dilemma  to  Air  Force  health  care 
executives.  Rising  training  costs,  inflation-eroded  training  budgets,  and  attrition  among  experienced  health 
care  profesiionals  are  expected  to  continue.  Substantially  increased  training  budgets,  protracted  training 
time  or  fewer  graduates  to  meet  medical  needs  are  unsatisfactory  solutions.  One  promising  alternative 
solution  cunently  in  use  in  some  academic,  governmental  and  industrial  settings  is  the  application  of 
cr-riiputer-assisted  instruction  to  instructional  areas. 

Computer-assisted  instruction  (CA!)  offers  the  opportunity  to  individualize  instruction  in  a tutorial 
matmer  such  that  instructional  objectives  ate  achieved  in  less  time  than  in  conventional  lock-step  modes  of 
instruction.  By  means  of  PLATO  IV  plasma  display  CAI  terminals  connected  to  a large  centralized 
computer  at  the  University  of  Illinois,  individual  students  can  respond  to  and  interact  with  instructional 
material  Immediate  analysis  of  student  performance  is  afforded  to  include  the  capability  of  quickly 
adapting  the  difficulty  level  or  type  of  presentation  approach  employed  for  an  individual  learner.  Aside 
from  the  motivational  impetus  generally  reported  by  students,  CAI  offers  standardization,  quality  control, 
and  objectivity  with  respect  to  instructional  objectives  attained. 

To  assist  informed  judgment  by  Air  Force  decision-makers,  preliminary  information  oa  the  suitability 
of  CAI  was  required  to  determine  if  subsequent  in-depth  study  of  the  comparative  effectiveness  of  CAI  as 
opposed  to  alternative  modes  of  conventional  instruction  was  warranted.  Whereas  the  present  report 
provides  preliminary  information  on  the  development  procedures.  Implementation  conditions,  student 
reactions  and  cost  data  analyzed  by  Air  Force  health  care  scientists,  the  second  phase,  currently  in  progress, 
emphasizes  the  systematic  and  objective  cwnpatison  of  CAI  with  alternative  instruction^  modes  on 
dimensions  of:  (a)  instructional  effectiveness,  (b)  completion  time,  (c)  delivery  costs,  (d)  student/instructor 
attitudes,  and  (e)  characteristics  of  learners  whose  performance-effectiveness  is  di^erentially  facilitated  or 
otherwise  influenced  by  alternative  instructional  modes. 

Preliminary  Endings  in  the  present  report  constitute  the  experiences  and  observations  of  the  PLATO 
IV  Development  Team  at  Sheppa^  AFB,  Texas.  Ihough  performance  and  attitudinal  data  are  customarily 
analyzed  by  appropriate  statistical  methods,  the  two  dis^ctly  separate  and  small  student  samples  (N^g, 
N=14)  did  not  permit  meaningful  statisticai  interpretation.  Qu^tative  analysis  of  student  performance  and 
attitudes,  which  included  student  interviews  performed  by  the  PLATO  IV  Staff  Clinical  Psychdogist,  forms 
the  basis  for  much  of  the  information  contained  herein. 

In  December  1973,  a formal  research  agreement  was  signed  by  representatives  of  the  School  of  Health 
Care  Sciences  (SHCS),  the  Advanced  Research  Projects  Agency  (ARPA),  and  the  Air  Force  Human 
Resources  Laboratory  ( AFHRL)  (Appendix  A).  This  agreement  tasked  the  ^hool  of  Health  Care  Sciences 
with  two  m^or  objectives: 

1.  The  research  and  development  of  a new  problem-oriented  medical  cui  lcuium  in  the  Physician 
Assistant  (PA)  course,  using  an  advanced,  computer-based  educational  system,  and, 

2.  A comparative  evaluation  of  the  instructional  effectiveness  of  this  curriculum  vis-a-vis  the  present 
traditional  curriculum. 

Impetus  for  origination  of  this  program  stemmed  from  a critical  shortage  of  physicians’  services,  both 
nationwide  and  in  the  USAF.  As  a result,  a new  category  of  health  specialist;  i.e.,  the  PA  course,  was 
developed,  like  most  of  the  civilian  PA  programs,  the  original  Air  Force  program  was  modeled  after  the 
traditional  medical  school  curriculum.  The  instructional  approach  was  characteristically  one  in  whidi  there 
was  a linear  sequence  of  basic  medical  sciences  which  temporarily  preceded  clinical  practice.  The  usual 
modes  of  instruction  and  learning  were  lectures/demonstrations  in  a group  lock-step  pattern. 

SHCS  officials  noted  that  the  major  problem  with  such  a curriculum  was  the  core  of  medical 
knowledge  which  must  be  memorized  is  usually  taught  without  a fimdan:ental  frame  of  reference,  ihat  is, 
basic  medical  sciences  are  often  presented  in  isolation  from  their  cliirical  relevance,  both  intellectudly  and 
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temporally.  Consequently,  the  basic  frame  of  reference  missing  in  the  traditional  PA  curriculum  was  a core 
of  clinical  applications  which  characterized  the  clinical  problem-solving  process  in  problem-oriented  medical 
contexts. 

During  the  time  of  the  original  research  proposal,  the  faculty  of  the  traditional  PA  course  had 
initiated  a comprehensive  planning  effort  which  examined  the  feasibility  of  converting  the  entire  PA  course 
to  the  problem-oriented  format.  This  effort  culminated  with  the  identification  of  24  “modules”  designed  to 
replace  the  traditional  PA  curriculum.  Had  this  total  conversion  been  pursued,  many  of  the  evaluation 
problems  detailed  elsewhere  in  this  report  would  have  been  avoided,  and  the  instmcUonal  impact  of 
computer-assisted  instruction  (CAI),  as  well  as  the  problem-oriented  medical  curriculum  (POMC)  could 
have  been  evaluated  with  more  confidence. 

Ihe  Hnal  research  agreement  tasked  the  School  of  Health  Care  Sciences  to  provide  (a)  adequate 
classroom  and  laboratory  space  to  accommodate  20  PLATO  IV  terminals  and  associated  hardware,  (b) 
provide  students  for  the  conduct  of  the  PLATO  evaluation,  (c)  -Provide  subject  matter  specialists  and 
technical  support  personnel  (Appendix  A)  to  produce  approximately  200  hours  of  PA  course  materials  for 
delivery  on  PLATO,  (d)  document  development  procedures  and  lessons  learned  for  future  reference,  (e) 
develop  a test  plan  for  the  evaluation  of  PLATO  IV  with  AFHRL,  and  (0  provide  monthly  progress  reports 
to  the  various  interested  agencies. 

To  insure  a common  frame  of  reference,  the  SHCS  PLATO  staff  defmed  a problem-oriented  medical 
curriculum  as: 

The  grouping  of  course  content  under  cU-  hai  problems  and  the  presentation  of  material  in  a clinically 
oriented  mode  such  that  the  clinical  relevance  of  the  course  content  is  stressed  in  order  to  develop 
problem.solving  skills,  patient  management,  and  investigative  skills.  Included  in  this  curriculum  are  certain 
basic  science  materiab  that  cannot  be  directly  related  to  clinical  problems.  This  material  will  be  included  in 
the  curriculum  as  prerequisites  to  bring  students  up  to  a minimal  acceptable  level  of  knowledge.  To  the 
maximum  extent  possible,  this  material  will  be  presented  in  such  a way  as  to  contrSrute  to  the  development 
of  problem-solving  skills  through  the  use  of  inquiry  learning  techniques. 

The  basic  POMC  concept  required  that  academic  subject  matter  not  be  introduced  as  isolated  units, 
and  emphasized  integration  into  the  context  of  clinical  diagnosis  of  actual  patient  problems.  Consequently, 
this  innovative  curriculum  drew  upon  1,400  hours  of  instructional  materia'  from  three  distinct  trimesters 
and  then  integrated  this  information  into  a problem-oriented  context.  The  new  POMC  course  material  was 
nut  an  hour-for-hour  replacement  of  traditional  instruction,  b it  rather  an  entirely  new  integration  of 
academic  information  and  skills.  Inherent  in  this  type  of  curriculum  was  the  notion  that  as  information  was 
disseminated,  the  individual  student  would  acquire  and  practice  diagnostic  problem-solving  skills. 
Ptoblem-solving  per  se  was  not  addressed,  but  the  use  of  patient  problems  and  simulations  served  as  a point 
of  reference  and  departure,  thereby  orienting  the  individual’s  thinking  toward  problem-solving. 

The  respiratory  system  was  selected  as  a feasible  area  for  instructional  development  of  135  POMC 
hours.  Ihe  rationale  behind  the  selection  of  the  respiratory  s>stem  and  total  POMC  hours  is  detailed 
ebewhere  in  thu  report. 

As  comprehensive  implementation  and  evaluation  plans  were  being  formulated  in  the  fall  of  1974,  a 
number  of  difflculties  surfaced.  For  example,  to  insure  sufricient  numbers  of  students  (N>50)  were 
exposed  to  the  POMC,  the  POMC  implementation  date  was  moved  forward  from  January  1976  to  October 
1975,  and  then  Anally  to  July  1975.  This  rescheduling  severely  compressed  the  lesson  development 
schedule,  but  did  not  result  in  any  student  delays.  Additionally,  serious  comparative  evaluation  problems 
(POMC  versus  traditional  curriculum)  surfaced  and  were  discus.sed  at  evaluation  conferences  during  October 
1974  and  March  1975  with  representatives  from  Hq  Air  Training  Command  (ATCX  AFHRL,  ARPA,  the 
Educational  Testing  Service  (ETS),  and  SHCS.  Ihe  major  obstacle  was  the  lack  of  a valid  and  reliable 
iiutrument  which  could  objectively  measure  tire  impact  and  effectiveness  of  the  POMC  which  constituted 
less  than  10  percent  of  the  1,400  PA  curriculum  houn  spread  over  a twelve-montl:  tseriod.  It  was  believed 
at  the  time  that  latent  image  simulations,  a feedback  technique  for  displaying  pro^'csses  and  outcomes 
resulting  from  student  answers  to  questions,  could  provide  such  a measu>e.  ETS  represei.tatives  pointed  out 
that  although  latent  image  simulations  were  used  on  many  national  board  type  examinations,  empirical 
evidence  of  their  ability  to  predict  clinical  success  was  lacking.  Inis  fact  was  disseminated  to  all  interested 
agencies  and  it  was  agreed  that  the  project  should  continue  since  it  would  still  generate  a substantial 
amount  of  attitudinal  and  “subjective”  data  concerning  the  POMC  and  computer-based  education  in 
general 
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The  evaluation  section  of  this  report  details  the  impact  of  implementation,  evaluation  strategies  and 
student  attitudes  on  the  outcome  of  the  project.  It  bvcame  feadily  apparent  during  November  1975  that 
the  impact  of  a number  of  factors  was  seriously  confounding  the  entire  POMC  research  eH'ort.  A thorou^ 
project  analyi  subsequently  dictated  a major  change  of  focus.  The  termination  of  the  POMC  project  and 
redirection  of  the  SHCS  PLATO  IV  research  effort  was  agreed  upon  and  finalized  during  Febru^  1976. 

The  foitjwi.ig  sections  provide  an  in-depth  examinaF.on  of  the  School  of  Health  Care  Sciences’ 
computer-based  FOMC  research  and  development  effo.t.  This  report  discusses  problems  encountered  and 
lessons  learned  for  future  reference. 


IL  INSTRUCTIONAL  MATERIALS  SELECTION  AND  DEVELOPMENT 

Subject-Matter  Selection.  The  following  considerations  v.ere  of  prime  concern  in  choosing  a 
sulqect-matter  area  for  t!',e  PO.MC: 

1.  Sulqect  matter  selected  had  to  be  fairly  circumscribed,  clinically  relevant,  and  capable  of  basic 
science  subject  matter  integration  since  these  characteristics  lent  themselves  to  a problem-oriented 
approach. 

2.  Subject  matter  selected  had  to  be  confined  to  approximately  10  percent  of  the  conventional 
year-long  PA  curriculum. 

3.  The  choice  of  topics  had  to  be  compatible  with  the  human  resources  and  equipment  with  whidi 
the  PLATO  IV  project  was  provided. 

Lesson  Material  Selection.  The  PA  curriculum,  like  other  Air  Force  training  courses,  was  well 
described  in  a Han  of  Instruction  (POI),  a document  outlining  the  subjects  taught  and  the  hours  devoted  to 
any  particular  portion  of  the  curriculum.  Thus,  for  any  of  the  possibilities  being  considered,  the  hours  of 
instruction  could  be  identified  and  easily  totaled. 

In  developing  the  prototype  mini-curriculum  on  the  “Resjriratory  System  and  its  Diseases,”  the 
project  had  to  insure  tliat  the  logical  flow  in  the  existing  discipline-oriented  approach  was  maintained. 
Practically  speaking,  this  seemed  difficult  because  the  current  PA  curriculum  consisted  of  a number  of 
indivisible  units  as  described  in  the  POI.  It  was  from  these  specified  blocks  of  material  that  the  subject 
matter  for  the  project’s  curriculum  was  extracted  and  subsequently  revised  into  a problem-oriented 
approach. 

Since  the  entire  body  of  knowledge  in  the  POI  had  to  be  taught  to  each  PA  student  during  the 
year-long  course,  the  PLATO  IV  project  staff  assumed  responsibility  for  teaching  individual  units  of 
respiratory  instruction  to  the  students  in  the  pilot  POMC  group.  POMC  students  were  to  cover  the  material 
in  PLATO  IV  lessons  in  lieu  of  the  lectures  their  classmates  would  be  attending.  Hus  resulted  in 
inconsistencies  in  project  curriculum.  For  example,  to  assume  responsibility  for  teaching  the  topics 
"Allergic  Rhinitis  and  Asthma”  to  the  pilot  group,  it  was  also  necessary  to  assume  responsibility  for  the 
entire  topic  of  “Ginical  Allergy,”  rather  than  just  a portion,  since  all  of  this  material  was  in  the  same  POI 
unit  and  was  inseparable.  In  this  case,  material  not  directly  pertaining  to  the  respiratory  system  was 
included  in  the  project’s  curriculum  so  that  relevant  topics  could  be  covered. 

In  contrast,  despite  the  interdependent  relatioiiships  between  certain  subject-matter  areas;  e.g., 
respiratory,  physiology  and  acid-base  balance,  relevant  material  could  not  always  be  neatly  extracted  from  a 
related  body  of  material  without  destroyitig  the  integrity  of  the  entire  subject  area.  It.  appeared  to  be 
beyond  the  scope  and  resources  of  the  project  to  cover  the  entire  topic  and,  consequently,  no  POI 
responsibility  was  taken  for  teaching  certain  basic  science  material  to  the  students. 

Basically,  the  material  extracted  from  the  POI  included  the  following  msgor  topics:  anatomy  of  the 
upper  airway,  chest,  and  lungs;  the  physiology  of  ventilation,  diffusion,  and  oxygenation  of  blood; 
interpretation  of  chest  X-rays;  the  pharmacology  of  antihistamines  and  antibiotics;  laboratory  staining  and 
culture  procedures;  diseases  of  the  nose  and  throat;  clinkal  allergy;  and  pulmonary  disease.  These  POI 
topics  are  enumerated  in  the  attached  document,  “POMC  Implementation  Plan,  10  February  1975” 
(/^pendix  B). 

Curriculum  Development.  Several  approaches  were  considered  for  curriculum  development.  Initially, 
it  was  felt  that  the  curriculum  should  be  composed  of  a series  of  descriptive  patient  encounters. 
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representing  the  gamut  of  respiiatoiy  disease.  each  of  these  descriptive  cases,  the  lesson  would  digress 
to  teach  the  appropriate  basic  science  material  (anatomy,  physiology,  microbiology). 

Because  the  prqect  had  to  cover  all  of  the  objectives  identified  in  the  POI,  and  s a result  of 
continuing  e.xperience  with  PLATO  as  an  instructional  tool,  the  initial  approach  wrs  sUgiuty  mcaified. 
Experience  with  PLATO  suggested  that  lesson  material  had  to  be  presented  in  a series  of  temporally 
manageable  modules  that  were  well  defined  and  limited  in  their  content.  It  was  felt  d.at  an  iriitial  basic 
science  overview  was  required  prior  to  exposure  to  the  bulk  of  the  curriculum.  This  base  scieno'  overview 
maintained  a clinical  orientation  whenever  possible,  with  references  to  the  clinical  appli'-a^'cns  of  various 
basic  science  principles. 

Fc^owing  the  initial  basic  science  overview,  a series  of  primarily  clinical  modules  \.'as  oteser  ted.  Each ' 
of  these  modules  dealt  with  a clinical  topic  in  pulmonary  disease;  whicb  reinforced  aiid  amplified  the 
related  basic  science  material. 

Simulated  patient  encounten  were  distributed  throu^out  the  POMC  sequence  of  lesson  material.  In 
these  situations,  the  student  gathered  sutqective  data  (history),  objective  data  (physical  and  lab  exam), 
assessed  the  information,  and  formulated  appropriate  plans.  These  simulated  patient  encounters  were 
designed  to  reinforce  systematic  techniques  of  data  collection  and  analysis,  as  well  as  to  serve  as  a vehicle 
for  teaching  and  reinforcing  clinical  science  subject  matter.  A sample  of  a typical  simulated  patient 
encounter  is  provided  and  more  fully  discussed  in  Appendix  D. 

Lesson  module  sequence  is  sliown  in  Figure  1.  POI  objectives  conesponding  to  each  block  in  the  flow 
chart  are  described  in  the  Implementation  Flan  (Appendix  B)  and  listed  in  greater  detail  in  Appendix  C. 

Lesson  Development.  Responsibility  for  developing  individual  modules  (‘‘lessons’'),  depicted  on  the 
flow  chart,  was  assigned  to  individual  subject-matter  experts  (“authors"),  and  lesson  developipcnt 
proceeded  as  follows. 

POI  Lesson  Objectives  The  objectives  stated  in  the  POI  for  a given  topic  were  identified  and 
reviewed.  In  many  cast  s,  the  objectives  were  not  sufficiently  precise  to  serve  a.s  lesson  objectives  or  to  serve 
as  the  framework  for  construction  of  test  items.  Examples  of  some  of  the  non-specific  “behavioral 
olyctives”  were: 

1.  Wock  XIII,  paragraph  3a:  Describe  the  anatomy  and  explain  the  physiology  of  tlie  respiratory 
system. 

2.  Hock  XXI,  paragraph  13d:  Discuss  mononucleosis. 

In  contrast,  some  of  the  objectives  in  the  POI  were  more  specific  which  facilitated  lessmt 
, .velopment  For  example: 

1.  Blodc  IX,  paragraph  4:  List  the  steps  in  the  Gram  stain.  Acid  Fast  Stain,  and  KOH  preparation, 
and  list  the  purposes  for  perfonning  microbiological  stains  on  clinical  preparations. 

The  imprecision  of  many  of  the  topic  objectives  in  the  PA  couise  was  primaiily  a reflection  of  the 
failure  to  carefully  define  the  PA  rule,  and  consequently,  the  content  of  much  of  the  PA  ‘mining. 

- - Accordingly,  there  was  tremendous  instructor  latitude  in  determining  the  breadth  and  depth  of 

presentation  for  any  given  topic,  especially  those  that  were  directly  clinical. 

For  example,  in  the  objectives  for  Block  XIII,  paragraph  3a,  there  were  no  guidelines  for  what  was 
important  and  hence,  what  shoiild  be  stressed.  Should  the  student  know  the  complex  sources  of  respirator 
control  in  the  brain  stem,  or  at  a lower  level  of  understanding,  was  it  sufficient  to  know  merely  that  the 
respiratory  drive  is  centered  in  the  brain  stem?  Responsibility  for  demonstrating  masteiy  of  instructional 
material  placed  instructors,  but  especially  students,  in  an  exceedingly  difficult  position,  given  the  ambiguity 
of  vague  POI  lesson  objectives.  How  was  the  student  to  know  which  materials  w jre  important  and  essential, 
and  which  were  not? 

The  most  reasonable  answer  was  to  present  subject  matter  which  the  instructor  regarded  as  important 
and  not  confuse  the  student  with  a deluge  of  peripheral  material.  Snee  the  PA  dinicai  role  was  evolving 
toward  increased  independence  and  responsibility,  there  was  a great  temptation  to  present  as  much  material 
as  possible  in  the  lectures.  Furthermore,  because  the  breadth  of  the  PA  professional  role  was  not  strictly 
defined,  students  attempted  to  absorb  as  much  as  they  possibly  could.  Students  also  fel‘  insecure  if 
someone  in  their  class  had  been  exposed  to  material  to  which  they  had  not.  Unfortunately,  students  lacked 
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the  professional  experience  and,  therefore,  the  discrimination  to  determine  what  subject  matter  was 
essential  and  wh?t  was  not.  In  many  cases,  because  of  vague  and  excessively  broad  topic  objectives, 
standardized  determination  was  not  made  for  them. 

PLATO  Lesson  Objectives.  After  reviewing  the  POI  objectives  for  the  identified  flow  chart  block,  the 
PLATO  author  developed  specific  instructional  objectives  for  the  particular  lesson.  The  objectives  were 
reviewed  by  the  instructional  system  development  (ISD)  specialist  for  clarity  and  adherence  to  Air  Force 
ISO  standards.  The  objectives  were  also  critiqued  by  a specific  associate  at  the  Center  for  Education 
Research  Laboratory  (CERL),  University  of  Illinois,  and  by  others  including  the  PLATO  staff  physician  and 
the  physician  assistant. 

In  the  early  stages  of  the  project,  the  process  was  often  quite  lengthy.  As  the  authors  became  more 
expert  in  writing  instructional  objectives,  the  review  procedures  became  less  time  consuming. 

Criterion  Test.  Once  the  behavioral  objectives  were  clearly  written  and  reviewed  to  delete  ambiguiaes, 
test  questions  were  written  so  that  initial  student  mastery  of  the  objectives  could  be  assessed  after  the 
lesson  was  completed.  Criterion  test  items  were  critiqued  by  the  CERL  Sheppard  east  (Shpeast)  as,^  iites, 
with  dialogue  carried  out  in  Sheppard  review  (Shprev).  Test  items  were  iso  evaluated  by  onsite  peer 
authors.  In  addition,  a consulting  group  at  the  University  of  Illinois,  consisting  of  members  of  the  CERL 
staff,  was  assigned  to  work  with  the  Sheppard  project  to  provide  lesson  reviews  and  technical  assistance  in 
programming. 

Author  and  Coder  Lesson  Roles.  Lesson  objectives  and  criterion  test  items  provided  the  basis  for 
lesson  development.  In  general,  the  author  who  wrote  the  script  for  the  lesson  also  coded  it  in  the  PLATO 
computer  program.  Several  attempts  weu  made  to  separate  the  roles  of  author  and  coder;  some  were 
successful,  wliile  some  were  not  successful  due  to  constraints  to  be  discussed  subsequently. 

The  series  of  lessons  comprising  the  pulmonary  embolism  sequence  was  written  by  the  PA  and  coded 
by  the  Medical  Service  Corps  officer.  This  arrangement  was  successful  because  of  the  physical  proximity  of 
the  individuals  involved  (working  in  the  same  office  and  able  to  sit  at  the  terminal  together).  In  addition, 
the  Medical  Service  Corps  officer  had  served  as  a hospital  administrator  and,  therefore,  was  familiar  with 
medical  procedures.  Therefore,  the  transcription  of  the  lesson  to  the  terminal  and  the  subsequent  editing 
were  able  to  proceed  without  communication  problems  and  with  a minimum  of  delay. 

The  chest  X-ray  sequence  of  lessons  was  a joint  effort,  with  the  script  written  by  the  PA  staff  member 
and  the  physician.  The  coding  was  shared  by  the  physician  and  the  Medical  Sc^ce  Corps  officer,  and 
development  proceeded  smoothly. 

Of  two  other  attempts  at  dividing  the  roles  of  author  and  coder,  only  one  was  partially  successful. 
The  prototype  simulated  patient  encounter  was  the  joint  effort  of  the  physician  and  an  associate  at 
Shpeast.  The  working  arrangement  was  successful  for  the  development  of  the  data  collection  sequence,  and 
the  expertise  of  the  individual  at  Shpeast  proved  invaluable.  For  the  balance  of  the  lesson,  where  less 
sophisticated  coding  was  required,  the  working  relationship  was  less  than  optimal  because  of  the 
communication  problems  involved  in  working  700  miles  apart. 

When  an  autlior  coded  his  own  script,  there  was  a great  deal  of  editing  during  the  transcription 
process  so  that  the  subject  matter  could  be  adapted  to  the  PLATO  medium.  Choices  of  display  techniques 
and  the  amount  of  material  on  a given  frame  greatly  affected  the  emphasis  given  to  any  pven  portion  of  the 
script.  These  editorial  choices  could  not  be  made  intelligently  unless  the  coder  had  a background  in  the 
medical  sciences.  Communicating  on-line  via  notes  or  “term-UJk”  proved  time  consuming  and  insufficient 
for  the  resolution  of  many  of  the  choices  to  be  made  during  cot^g.  In  addition,  since  the  lessons  were 
being  valid.itcd  at  Sheppard,  it  proved  much  more  efficient  for  the  author  to  be  able  to  edit  them  quickly, 
ratlier  than  transmit  comments  to  the  coder  700  miles  away,  particularly  with  die  inherent  delays  in 
implementation  ard  problems  in  communication. 

The  other  attempt  at  separating  the  roles  of  author  and  codei  occuned  in  the  Pulmonary  Function 
Studies  lesson.  Since  little  sophisticated  coding  was  required,  it  became  apparent  very  early  that  the 
shortcomings  noted  above  far  outweighed  freeing  the  author  of  coding  responsibilities.  In  essence,  it  proved 
impossible  to  separate  the  content  from  the  medium.  Unless  the  coder  and  author  were  actually  working 
together  at  the  same  site  and  unless  the  coder  also  had  some  expertise  in  the  subject  matter,  it  was  much 
more  beneficial  to  combine  the  roles  of  author  and  coder. 
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Lesson  Review  and  Validation.  Prior  to  completion  of  entire  lessons,  completed  segments  were 
reviewed  by  associates  at  Shpeast  and  by  author  peers  at  Sieppard.  Lesson  review  was  both  formal  and 
informal  (oral  comments  and  casually  written  impressions).  Formal  review  by  associates  at  Shpeast  was 
generally  a formal  critique  recorded  in  the  lesson  spaee  “Shprev.”  Informal  review  by  peers  at  Sheppard  was 
generally  oral,  supplemented  by  casually  written  notes.  These  casually  written  notes  were  discarded  after 
they  were  read  and  suggested  changes  were  made. 

Student  Review.  After  initial  reviews  at  the  author  level,  lessons  were  revised  appropriately  and  initial 
trial  evaluation  commenced.  Lessons  were  admint  tered  on  a trial  basis  to  several  groups  of  students. 

One  group  consisted  of  four  early  arrival  PA  students  who  completed  all  of  the  available  lesson 
material.  Orta  were  collected  on-line  and  included; 

1 . Time  required  to  complete  an  entire  lesson. 

2.  Time  required  to  complete  a ^ven  segment  of  a lesson. 

3.  Responses  to  interactive  lesson  material. 

a.  Number  of  attempts  required  to  correctly  answer  a question. 

b.  Incorrect  answers  that  had  been  anticipated  by  the  author. 

c.  Inconect  answers  that  had  not  been  anticipated  by  the  author. 

4.  Number  of  requests  for  assistance  (-HELP—). 

5.  Performance  on  the  concluding  quiz  to  assess  immediate  mastery  of  the  material. 

The  data  were  collected  in  a data  file  and  subsequently  analyzed.  Where  —HELP—  was  sought  by  the 
student  and  not  available,  it  was  added  if  it  were  felt  by  the  author  that  it  * 'ould  enhance  the  lesson. 
Interactive  material  was  edited  to  incorporate  answers  that  had  been  unanticipateu  so  that  specific  feedback 
could  be  given  for  wrong  answers. 

COMMENT:  Despite  concurrent  relative  instability  of  the  system  (transmission  errors  and  system 
“crashes”),  the  students  displayed  a very  high  threshold  for  frustration.  Their  verbal  comments  reflected  a 
high  degree  of  interest  both  in  the  system  and  the  individual  lessons.  It  must  be  noted  that  the  students 
were  under  no  other  academic  pressures  because  their  classes  had  not  started  and  they  displayed  a ve.’ 
relaxed  attitude.  Despite  a favorable  response,  none  of  the  students  returned  frequently  to  review  or  tak; 
new  lessons  during  the  semester.  The  students  stated  that  the  pressures  and  time  demands  of  the  PA 
curriculum  prevented  their  returning  on  a frequent  basis. 

Review  by  Other  Medical  Personnel.  Lessons  were  also  taken  by  personnel  who  worked  in  medical 
areas  but  who  were  not  PA  students.  This  group  included  nurses,  laboratory  and  pharmacy  technicians,  as 
well  as  hospital  corpsmen.  Some  of  these  individuals  were  stationed  at  Chanute  Air  Force  Base,  Illinois. 

The  same  data  were  collected  on-line  from  these  individuals  as  were  collected  from  the  trial  group  of 
PA  students.  Comments  were  made  on  an  informal  basis  regarding  the  mode  and  clarity  of  presentation,  as 
well  as  any  perceived  inaccuracies  in  the  lesson  material. 

COMMENT:  Formal  and  infonnal  data  from  all  of  these  pilot  studies  indicated  that: 

1.  The  PLATO  medium  was  well  received  and  perceived  by  the  student  an  an  effective  instructional 
medium. 

2.  Transmission  errors  and  system  “crashes”  at  this  time  caused  a minimum  of  frustration. 

3.  Coding  problems  in  individual  lessons  were  not  a m^or  source  of  frustration. 


in.  STUDENT  SAM?1.E 

Sample  Selection.  A random  selection  process  was  use^  for  assigning  students  to  either  the  PLATO  IV 
or  lecture  group  conditions.  Each  Navy  PA  student  and  Ailr  Force  PA  student  was  assigned  a number  and 
numbers  were  randomly  drawn  from  the  population  pool  for  asagnment  to  one  of  the  conditions. 
Statistical  analysis  of  historical  data  indicated  that  within  the  PA  program  the  only  variable  with  any 
alleged  success  of  prediction  potential  for  completion  of  course  was  the  Scholastic  Aptitude  Tests  (SAT) 
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score.  Li  the  first  class  (June  197S),  the  mean  SAT  score  for  the  Air  Force  PLATO  IV  group  was  1062.5 
(S.D.  1 11.7),  while  the  lecture  control  group  mean  SAT  score  was  1009  (S.D.  149.5).  Naval  persoruiel  did 
not  have  SAT  scores,  but  rather  provided  similar  aptitude  scores  on  Navy-developed  General  Aptitude 
Tests.  The  Navy  PLATO  IV  group  mean  score  was  127.0  (S.D.  9.2),  wliile  the  Navy  control  group  mean 
score  was  128.7  (S.D.  7.1).  No  statistically  agnificant  differences  existed  on  these  a priori  measures 
between  the  PLATO  IV  and  lecture  groups.  In  the  second  class  (November  1975),  the  Air  Force  mean  SAT 
score  for  the  PLATO  IV  group  was  1 148  (S.D.  124.6),  while  the  mean  SAT  score  for  the  control  group  was 
1 100  (S.D.  1 15.5).  The  mean  SAT  scores  of  the  control  and  PLATO  IV  groups  were  analyzed  using  the 
t-test  and  no  significant  difference  was  found.  Naval  aptitude  scores  were  no,  available  for  Navy  personnel 
involved  in  the  second  class.  In  both  classes,  the  proportion  of  Air  Force  and  Navy  students  was  not 
significantly  different  from  the  ratio  of  Navy  and  Air  Force  students  in  the  total  PA  program.  This  sampling 
process  was  in  concurrence  with  the  guidelines  set  forth  in  the  sampling  considerations  portion  of  the 
attached  Evaluation  Plan  (Appendix  E). 

Student  Oiaracteristics.  Many  unique  characteristics  of  the  PA  curriculum  and  the  PA  students 
should  be  discussed  in  order  to  understand  the  target  population  and  the  academic  environment  in  which 
they  worked.  First,  the  average  age  of  the  military  PA  student  was  28  years.  Student  ages  tanged  from  23  to 
37  years,  most  were  married,  and  had  been  on  active  duty  for  four  to  tiurteen  years.  Most  of  the  students 
were  corpsmen  with  considerable  experience  in  the  medical  field. 

The  military  PA  education  program  differed  in  many  ways  from  the  PA  programs  at  civilian 
institutions.  In  the  military  program,  the  students  attended  class  (5-day  week)  from  0800  to  1700  daily. 
There  were  no  elective  courses  so  each  class  was  an  integral  part  of  liis  education  and  was  material  for 
testing.  After  the  eight  hours  of  solid  class  work,  the  students  studied  in  groups  for  three  to  five  hours  each 
night  and  four  to  eight  hours  each  weekend.  All  classes  were  mandatory  so  there  were  no  “cuts”  to  catch 
up  for  an  examination  or  prepare  for  another  class.  Tliere  were  usually  one  or  two  block  examinations  each 
week,  plus  at  least  one  physical  examination  report  each  week  for  clinic  patients.  Students  were  in  an 
extremely  rigid,  higli-pressure,  lock-step  regimen.  The  students  had  two  one-week  breaks  between  the 
trimesters,  which  were  used  to  fulfill  military  training  requirements,  tend  to  personal  business,  and  ta“ke 
leave  if  desired. 

With  all  these  pressures  and  the  long  hours  of  intense  study,  why  did  people  apply  for  the  PA  school 
and  continue  to  work  when  they  were  accepted?  Basic.ally,  these  mature  individuals  were  very  ambitious 
and  highly  competitive.  The  rewards  for  completing  the  course  included  more  responsibility  in  their  chosen 
field  of  medicine,  increased  pay  and  rank,  acquisition  of  college  credits  through  the  University  of  Nebraska 
which  could  lead  to  a baccalaureate  degree,  and  training  which  was  useful  for  them  in  a civilian  community 
when  they  retired.  All  of  these  positive  incentives  to  succeed  were  reinforced  by  several  strongly  negative 
aspects  if  tliey  did  not  succeed. 

If  a student  was  unable  to  carry  a passing  average  in  any  major  subject  block,  an  academic  board  was 
convened.  If  the  board  felt  the  student  could  continue,  he  was  given  a makeup  examination.  If  this 
examination  was  passed,  the  student  continued  his  normal  course.  If  he  failed  the  retest  there  were  two 
alternatives.  First,  the  student  was  recycled  to  begin  the  entire  trimester  for  a second  time.  Tliis  was 
necessary  because  tliere  were  no  provisions  for  a student  to  receive  remedial  training  in  any  single  area.  If 
the  student  was  recycled,  it  required  rescheduling  Riase  II  training,  adjusting  family  plans  for  transfer,  and 
resulted  in  many  complications  to  the  plans  of  tlic  Air  Force  and  the  student. 

If  the  student  failed  the  retest  and  the  academic  board  did  not  feel  the  student  could  succeed  in  the 
PA  course,  he  was  subject  to  elimination.  This  failure  was  not  of  major  trauma  to  the  student’s  military 
career,  but  usually  had  many  negative  connotations  to  the  student  himself.  Being  highly  competitive,  these 
students  perceWed  failure  as  a reflection  on  their  capabilities.  Being  very  ambitious,  they  also  re.ilized  that 
by  not  completing  the  course,  they  did  limit  their  potential  to  function  in  a highly  responsible  role  in  the 
field  of  military  medicine. 

Considering  all  of  these  factors,  it  became  obvious  that  there  were  a multitude  of  pressures  inherent 
in  the  academic  environment  which  were  accentuated  by  the  compulsive,  ambitious,  competitive 
characteristics  of  the  students.  Tlic  students  were  constantly  tense  and  anxious,  and  they  had  a low 
frustration  tolerance  for  unanticipated  problems  or  extra  demands  upon  tlieir  inflexible  schedules.  Tlie 
effects  of  this  low  resistance  to  frustration  upon  the  acceptance  of  the  PLATO  IV  project  will  be  discussed 
at  length  in  the  Student  Attitude  Survey  section. 
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IV.  IMPLEMENTATION 


One  of  the  major  tasks  confronting  the  SHCS  PLATO  staff  was  to  provide  for  the  orderly  integration 
of  the  POMC  into  the  traditional  PA  curriculum.  Ideally,  this  integration  would  have  been  done  in  such  a 
way  that: 

1.  Disruption  of  the  traditional  cuniculcm  was  minimized. 

a.  Steps  had  to  be  taken  to  insure  that  prerequisite  material  was  taught  prior  to  the  time  it  was 
required  as  a basis  for  other  material. 

b.  The  logical  flow  of  content  in  discipline-oriented  blocks  of  instruction  was  not  jeopardized. 

2.  The  integrity  of  the  POMC  was  maintained;  i.e.,  presentation  of  material  was  not  excessively 
fragmented. 

3.  No  degradation  of  training  would  result. 

4.  Evaluation  of  the  POMC  was  facilitated. 

In  attempting  to  develop  implementation  plans  against  which  the  above  criteria  could  be  applied, 
several  alternatives  were  considered.  These  alternatives  included  the  following  possibilities: 

1.  Extension  of  the  course  so  that  inclusion  of  POMC  materials  would  require  no  changes  in  the 
conduct  of  the  course.  Under  this  approach,  students  selected  for  the  test  group  would  have  been  brought 
to  the  school  about  four  weeks  before  the  start  of  their  normal  PA  class  to  complete  the  POMC  lessons 
prior  to  beginning  the  traditional  curriculum.  This  approach  was  suggested  by  the  PA  course  director,  but 
was  not  considered  to  be  feasible  by  the  PLATO  staff  for  the  following  reasons: 

a.  Since  only  one  module  of  the  POMC  was  being  developed,  certain  prerequisite  material  was 
still  contained  in  the  traditional  curriculum  and,  therefore,  students  could  not  be  exposed  to  the  POMC  in 
isolation  from  the  traditional  curriculum. 

b.  A comparative  evaluation  would  not  have  been  possible  because  a control  group,  parallel  to  a 
POMC  test  group,  would  not  have  existed  under  this  approach. 

c.  Extension  of  tlie  course  could  not  be  accommodated  within  Hq  ATC  budgetary  and  training 
day  restrictions. 

2.  Another  alternate  approach  considered  was  to  create  an  additional  block  of  instruction  in  the 
second  or  third  trimester  in  which  the  POMC  material  would  be  concentrated,  with  hours  for  this  additional 
block  coming  from  several  different  traditional  blocks.  Under  this  approach,  m^or  reorganization  of  the 
course  would  have  been  required.  Units  of  instruction  would  have  had  to  be  moved  from  one  semester  to 
another,  and  the  basic  science  lectures,  in  particular,  would  have  had  to  be  restructured.  Because  of  the 
class  size  (64  students)  and  the  number  of  PLATO  terminals  available,  the  approach  would  also  have 
required  either  the  development  of  lectures  comparable  to  the  computer-based  POMC  lessons  or  a hi^ly 
complex  scheduling  effort,  with  lectures  being  repeated  as  groups  alternated  between  the  PLATO  classroom 
and  the  traditional  classroom,  which  posed  an  unacceptable  lecturer  manpower  problem. 

3.  At  the  time  the  request  for  personnel  research  was  submitted,  considerable  effcii  had  been 
expended  on  preliminary  planning  to  convert  the  entire  PA  course  to  a series  of  problem-oriented  modules 
without  computer-based  instruction.  With  the  inception  of  the  PLATO  project  and  throughout  the 
discussions  regarding  implementation  of  the  PLATO  POMC  module,  no  further  steps  toward  this  possibility 
were  entertained.  This,  in  'arge  part,  was  due  to  a change  in  course  management  and  an  almost  total 
turnover  in  the  physician  staff  assigned  to  the  conventional  course. 

A compromise  was  reached  whereby  the  POMC  module  developed  by  the  PLATO  staff  was 
implemented  in  the  following  way. 

On  the  basis  of  criterion  objectives  listed  in  the  POI,  specific  hours  of  instruction  to  be  included  in 
the  POMC  were  identified.  Students  in  the  test  group  were  to  complete  PLATO  materials  in  lieu  of  the 
cor.  oponding  work  in  the  traditional  curriculum.  They  attended  all  of  the  scheduled  lectures,  except  those 
that  had  been  replaced  by  PLATO  lessons.  Considerable  concessions  were  made  in  the  selection  of  material 
included  in  the  POMC  so  that  discrete  lecture  hours  of  the  traditional  cuniculum  were  identified.  At  the 
time  these  lectures  were  given,  PLATO  students  left  the  classroom  and  reported  for  computer-based 
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instruction.  Overall,  the  same  criterion  objectives  were  to  be  met  by  all  students,  but  the  temporal 
sequencing  of  POVC  differed  from  that  of  the  traditional  curriculum.  To  some  extent,  the  sequencing  of 
POMC  materials  was  dictated  by  the  traditional  curriculum  because  of  the  problem  of  insuring  that 
prerequisite  material  was  taught  prior  to  the  time  it  was  required  in  the  traditiorial  curriculum,  even  if  this 
material  was  a part  of  the  POMC.  For  example,  in  the  problem-oriented  approad'.  lessons  on  sputum 
processing  and  staining  procedures  would  normally  have  been  taught  in  conjunction  with  material  on 
infectious  lung  disease,  but  because  of  a staining  laboratory  exercise  early  in  the  traditional  curriculum, 
these  lessons  were  presented  during  the  second  scheduled  hour  on  PLATO. 

In  examining  the  criteria,  listed  previously,  with  regard  to  «he  implementation  scheme  finally 
adopted,  it  is  clear  mat  the  preponderance  of  effort  was  made  to  meet  the  first  criterion,  that  of  minimizing 
disruption  of  the  traditibnal  curriculum.  Attention  was  also  given  to  insure  that  “no  degradation  in 
training”  was  also  met,  but  tliis  was  never  thought  to  be  a strong  possibility  under  any  implementation 
considered.  Criteria  regarding  maintaining  the  integrity  of  the  POMC  and  facilitating  its  evrduation  were 
compromised  to  obtain  “minimized  disruption  of  the  curriculum.”  The  only  milestones  contained  in  the 
research  agreement  were  the  start  date,  completion  date,  and  final  report  date.  An  implementation  date  of 
January  1976  had  been  assumed  by  members  of  the  PLATO  staff  in  the  belief  that  six  months  would  be 
adequate  for  the  evaluation  phase  of  the  project,  thereby  aUowing  the  bulk  of  the  project  time  to  be 
devoted  to  curricular  design  and  lesson  development.  Since  the  POMC  was  to  be  im^emented  over  the 
entire  year-long  PA  course  rather  than  being  concentrated  into  a segment  of  the  course,  it  became 
imperative  that  implementation  and  evaluation  begin  at  the  earliest  possible  data;  i.e.,  June  1975.  This  date 
enabled  administration  of  the  curriculum  to  two  classes  for  evaluation  and  preparation  of  the  final  report. 
Implementation  was,  in  fact,  accomplished  with  the  June  1975  class,  but  the  expected  time  «!vailable  for 
lesson  development  was  reduced  by  six  months. 

The  POI  hours  from  the  traditional  course  included  in  the  POMC  were  distributed  over  the  three 
trimesters  of  the  PA  course  as  stated  in  Table  1 ; 


Table  1.  Traditional  Course  Hoiin 
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POI  CUM/CTT*  Moun 

1st 

14/4.5 

2nd 

15/3.0 

3rd 

50/20 

79/27.5 

^Student  Homework 


In  the  first  trimester,  the  fouiteen  hours  of  lecture  (replaced  by  PLATO  material)  were  scattered 
across  the  first  thirteen  weeks,  with  as  much  as  four  weeks  intervening  between  PLATO  sessions.  A detailed 
distribution  of  hours  is  shown  in  Table  2.  It  should  be  noted  that  all  PLATO  sessions  were  either  one  hour 
or  two  hours  long  with  never  more  ihar,  one  ^LATO  session  per  day.  In  the  second  trimester,  the  hours 
replaced  were  primarily  concentrated  in  weeks  seven,  eight,  and  nine,  so  that  some  continuity  within  the 
POMC  was  obtained.  Table  2 also  shows  the  detailed  distribution  of  second  trimester  hours. 


Table  2.  Distribution  of  Plato  Hours  by  Trimester/by  Week 
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Since  students  in  the  PLATO  group  were  not  working  on  the  same  subject  matter  at  the  same  time  as 
the  rest  of  the  class,  some  modification  of  the  normal  end-of-block  criteria  measurement  procedures  was 
required.  A waiver  of  the  proviaons  of  ATCR  52-3  was  applied  for  and  granted  by  Hq  ATC  so  that  the 
procedures,  described  next,  could  be  implemented  regarding  tesimg 

1.  Prior  to  administering  an  examination  or  quiz  which  covered  objectives  for  which  PLATO 
materials  were  involved,  the  PA  faculty  member  planning  to  administer  the  measurement  instrument 
contacted  his  PLATO  staff  counterpart.  Jointly,  the  PLATO  staff  member  and  PA  faculty  member 
determined  which  questions  were  to  be  exduded  from  tlie  examination  for  PLATO  students. 

2.  At  the  same  time  of  the  examination,  PLATO  students  were  directed  not  to  answer  the  questions 
identified  by  the  previous  procedure.  The  instructor  was  to  write  the  numbers  of  the  questions  to  be 
excluded  on  the  blackboard  and  have  the  students  in  the  PLATO  group  circle  those  numbers  on  their 
answer  sheets. 

3.  At  the  time  the  examination  was  graded,  percentage  scores  were  calculated  for  PLATO  students 
on  the  basis  of  the  number  of  items  answered  conectly  out  of  the  subset  of  items  administered.  For 
example,  if  five  questions  were  excluded  on  a test  which  normally  contamed  fifty  questions,  the  PLATO 
student  score  was  the  number  answered  correctly  divided  by  forty-five. 

4.  The  PLATO  staff  was  responsible  for  testing  objectives  covered  by  PLATO  material.  Formal 
PLATO  tests  were  administered  and  the  results  of  these  tests  were  used  to  adjust  both  raw  scores  and 
percentage  scores  of  tests  from  which  questions  were  excluded.  The  following  procedures  were  used  in 
adjusting  these  scores. 

a.  A weighting  factor  was  determined  for  each  test  in  the  regular  curriculum  that  was  affected  by 
the  adjustment  procedures.  For  example,  a test  which  normally  contained  50  questions  from  which  eight 
questions  were  exduded,  had  a weighting  factor  of  .84.  The  weighting  factor  of  the  PLATO  test  was 
ftcrefore  .16,  such  that  the  sum  of  the  weighting  factors  equaled  1.00. 

b.  At  the  completion  of  a PLATO  testing  cycle,  all  affected  test  scores  (percentage)  in  the 
regular  curriculum  which  had  not  been  adjusted  previously  were  adjusted  according  to  the  following 
formula. 

Test  Wei^ting  Raw  PLATO  Wei^ting  Plato  Adjusted  Adjusted 

Factor  x Score  + Factor  x Score  = Test  Score 

Raw  scores  were  calculated  by  the  formula: 

Adjusted  Total  Questions  on  Adjusted 

Test  Score  x Test  Before  Exclusions  = Raw  Score 

An  example  of  testing/grading  procedures  used  for  PLATO  students  is  shown  in  Appendix  F. 

Impkmentatjon  Constraints.  Many  of  the  problems  discussed  in  the  evaluation  chapter  of  this  report 
regarding  student  attitudes  had  their  origin  in,  or  at  least  were  exacerbated  by,  implementation  constraints 
which  resulted  in  student  reaction  to  differences  in  the  material,  when  material  was  provided,  and  how 
PLATO  students  were  tested  compared  to  conventional  students.  For  example,  POMC  material  was 
intermittently  diffused  across  three  trimesters  with  sessions  scheduled  based  upon  available  hours  as 
dictated  by  the  traditional  PA  class  schedule.  The  continuity  of  the  POMC  was  severely  disrupted  as  a 
result  Fragmentation  of  the  curriculum  was  so  excessive  that  familiarity  with  terminal  operation  was  never 
fully  achieved  such  that  students  were  not  completely  at  ease  and  comfortable  with  the  system. 

Another  key  implementation  issue  with  students  dealt  with  competing  time  schedules  between 
lecture  and  PLATO  conditions.  Much  of  the  self-pacing  aspect  of  PLATO  was  lost  or  severely  confined,  due 
to  the  schedule  within  the  traditional  classroom  lecture  to  which  the  POMC  student  returned  on  time 
regardless  of  POMC  lesson  completion.  Lessens  not  completed  during  a scheduled  PLATO  session  had  to  be 
completed  either  at  the  next  session,  which  could  be  a considerable  time  later,  or  on  the  student’s  own  time 
during  lunch  hour  or  after  normal  duty  hours. 


V.  EVALUATION  PLAN 

As  a result  of  several  meetings  with  ARPA,  AFHRL,  and  ATC,  agreement  was  made  to  evaluate  three 
primary  areas.  The  first  area  was  academic  effectiveness.  It  was  important  to  insure  that  the  introduction  of 
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the  POMC  was  not  accomplished  at  the  expense  of  overall  academic  understanding  of  the  subject-matter 
content.  Indeed,  the  intention  of  the  POMC  was  to  promote  a greater  understanding  of  content  through  an 
organized  problem-oriented  approach.  To  evaluate  the  instructional  effectiveness  of  the  POMC  in  teadiing 
respiratory  disease,  it  was  necessary  to  develop  a common  academic  test  which  could  be  given  to  both  the 
experimental  and  control  groups  at  the  end  of  the  fust  year  training  (Study  1,  ^pendix  E).  The  test 
covering  respiratory  disease  was  to  be  constructed  using  items  that  were  sin^ai  to  those  found  on  the 
American  College  of  Medicine  board  examinations.  The  pool  of  items  for  this  examination  was  to  be 
obtained  through  combined  contributions  of  several  civilian  universities  that  were  sponsoring  PA  training. 
To  eliminate  bias,  questions  on  respiratory  disease  from  other  PA  programs  uere  considered  the  most 
appropnate  source  for  items  since  the  student  populations  were  essentially  the  same.  Once  this  pool  of 
questions  was  refined,  a group  of  PA  students  who  had  completed  Phase  I of  the  two-year  program  were  to 
Im  given  the  test  for  standardization  purposes.  Item  analyses  were  planned  to  statistically  insure  that  only 
items  which  distinguished  trained  from  Untrained  students  would  be  used  for  the  final  evaluation 
instrument  The  refmed  version  of  this  test  was  to  be  administered  to  both  the  test  (POMC)  and  control 
(traditional  lecture)  groups  at  the  end  of  the  1 ,400-hour  Phase  1 training  program. 

The  second  area  to  be  evaluated  was  that  of  clinical  problem-solving  skQls.  Simulated  patient 
encounters  were  to  be  used  to  lest  the  students’  ability  to  integrate  their  knowledge  toward  successful 
dia^osis  and  treatment  of  patients.  Ihese  instruments  were  developed  through  collaborative  efforts 
between  the  SHCS  PA  staff,  the  PLATO  staff,  and  the  ETS.  These  tests  were  designed  to  be  interactive 
patient  simulations  which  would  allow  the  students  to  request  information,  obtain  histories  and  lab  results, 
prescribe  treatment,  and  select  diagnoses.  Simulations  included  not  only  patient  simulations  for  respiratory 
disease,  but  also  included  several  other  patient  amulations  in  the  area  of  internal  medicine.  The  inclusion  of 
two  types  of  simulations  was  intended  for  comprnsons  of  general  problem-sdving  skills  and  specific 
problem-solving  skills  in  the  area  of  respiratory  diseases. 

The  third  major  area  of  evaluation  was  to  determine  what  effect  the  POMC  and  the  use  of  the  PLATO 
system  had  on  students’  attitudes  toward  learning.  (See  Study  3,  Appendix  E).  Toward  this  end,  a 30-item 
revision  of  the  Brown  questionnaire  (1968)  composed  of  Likert-type  items  was  selected  as  an  attitude 
measure  (Appendix  G).  This  instrument  had  demonstrated  reliabflity  (r=.80)  with  college  students  in 
computer-based  education  and  included  many  specific  questions  which  permitted  inferences  about  vaious 
features  of  the  PLATO  system.  In  addition  to  the  Likert-scaled  items,  there  were  two  open-ended  questions 
which  were  intended  to  elicit  specific  comments  about  areas  not  covered  by  the  30  questions  and 
explanations  of  responses  in  the  initial  portion  of  the  questionnaire.  Analyses  of  the  relationship  between 
attitude  and  performance  on  the  final  academic  test  and  the  patient  simulations  were  planned. 

In  addition  to  the  end-of-first-year  evaluation  of  academic  effectiveness,  a study  was  plaimed  to 
determine  if  POMC  students  were  meeting  learning  objectives  at  the  lesson  level  as  they  progressed  through 
the  course.  This  evaluation  was  to  be  achieved  by  determining  the  number  of  lessons  repeated,  f'e  number 
and  duration  of  special  or  remedial  sessions,  and  the  number  of  eliminations  and  washbacks.  Relationship 
between  student  achievement  at  the  lesson  objective  level  (and  subsequently  at  die  block  level)  were  also 
deemed  an  appropriate  index  of  evaluation. 

In  addition  to  evaluation  of  student  attitudes  and  progression,  there  was  considerable  interest  in 
determining  CAI  author  characteristics.  If,  in  the  future,  computer-based  education  is  to  become  an 
operational  form  of  military  training,  information  on  criteria  for  CAI  author  selection  will  be  needed. 
Several  qualitative  areas  for  identification  of  author  characteristics  were  reviewed.  First,  evaluation  of 
authr'r  performance  was  considered.  Author  performance  was  to  be  measured  in  three  ways: 

a.  Average  quality  of  lesson  produced  as  judged  by  the  University  of  Illinois  penonnel  and  POMC 
students. 

b.  Supervisory  ratings  of  author  performance. 

c.  Peer  ratings  of  author  performance. 

Performance  dimensions  to  be  rated  were  to  include  programming  ability,  efficiency  in  lesson  design, 
creative  use  of  instructional  techniques  and  novel  applications  of  the  PLATO  system. 

An  evaluation  of  author  attitudes  was  also  planned.  The  purpose  of  this  study  was  to  determine  the 
relationship  between  the  authors’  attitudes  toward  CAI  and  their  performance  as  effective  authors.  Because 
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of  the  small  number  of  actual  authors  in  the  sample,  statistical  analysis  of  information  was  not  considered, 
but  subjective  analyses  of  various  areas  of  evaluation  were  possible  and  practical. 

The  last  study  to  be  accomplished  with  the  authors  was  an  attempt  to  relate  author  personal  and 
background  characteristics  to  performance  on  the  PLATO  project.  Toward  this  end,  data  were  collected  on 
such  variables  as  educational  level,  age,  rank,  IQ,  attitudes,  volunteer  versus  nonvolunteer  status,  and 
interest  in  computer-based  education.  The  aim  of  this  study  was  to  develop  a set  of  criteria  for  selection  of 
future  PLATO  authors. 

The  final  portion  of  the  evaluation  plan  was  to  provide  recommendations  regarding  the  effectiveness 
of  PLATO  as  a teaching  medium  The  PLATO  system  was  to  be  evaluated  in  terms  of  instructional 
effectiveness,  overall  costs  and  reli  oility.  Basic  cost  data  were  available  through  the  Resource  Management 
Divisiot.  of  the  School  of  Health  Care  Sciences  and  through  capital  investment  figures  obtained  from 
ARPA.  System  reliability  was  to  be  evaluated  by  means  of  system  logs  which  were  to  be  maintained  to 
show  the  number  of  terminals  in  commission  per  day,  percentage  of  down  time  per  terminal,  and  cause  of 
equipment  failure. 


VL  STUDENT  ATTirUDE  SURVEY 

One  of  the  major  objective,  if  the  student  attitude  survey  was  to  evaluate  students’  attitudes  toward 
the  POMC  and  PLATO  IV  as  a m'  aium  of  instmction.  The  attitude  questionnaire  was  administered  after  at 
least  one  month  of  exposure  to  PLATO  coursework  to  obtain  initial  sentiments.  Retest  was  to  be 
accomplished  at  the  end  of  the  entire  curriculum,  which  permitted  comparisons  to  be  made  on  a one-year 
exposure.  In  the  original  evaluation  plan,  it  was  proposed  that  all  of  the  attitude  surveys  for  both  POMC 
PLATO  classes  be  combined  and  the  combined  data  used  to  indicate  PA  attitudes  toward  the 
POMC-PLATO  condition.  Termination  of  the  POMC  and  redirection  of  the  PLATO  project  to  courses  more 
suitable  to  a comparative  analysis  of  PLATO  and  conventional  instruction  therefore  precluded  the 
opportunity  to  conduct  the  one-year  test-retest  of  attitudes.  Consequently,  analysis  of  students’  attitudes 
was  confined  to  separate  examinations  of  individual  dass  attitudes  in  the  two  PLATO  test  groups. 
Discussion  of  the  attitudinal  differences  between  the  two  classes  as  they  related  to  experience  with  the 
POMC  specifically,  and  PLATO  IV  in  general,  follows  the  separate  analysis. 

Class  1.  Initial  attitude  testing  of  the  first  PA  group  demonstrated  a rather  large  number  of  neutral  to 
mildly  negative  responses  toward  PLATO  CAI.  Many  of  the  negative  responses  were  related  to  system 
malfunctions  or  to  technical  programming  difficulties  with  individual  lessons.  The  difficulty  with  individual 
lessons  was  generally  related  to  the  lack  of  smooth  sequencing  through  large,  Comdex  lessons.  On 
occasions,  students  encountered  areas  where  they  could  not  proceed  or  were  inadvertently  rerouted  to  the 
beginning  of  a lesson  rather  than  continuing  after  a —HELP—  sequence.  The  area  of  major  dissatisfaction 
was  the  number  of  system  malfunctions  that  consistently  were  present.  Although  there  were  no  major 
system  problems  of  down  time  with  which  the  students  had  to  contend,  the  minor  irritation  of  transmission 
line  errors  which  obscured  legibility  appeared  to  have  a cumulative  effect  and  became  a major  irritant  to  the 
students. 

The  favorable  comments  toward  CAI  were  primarily  related  to  the  ability  to  progress  through  the 
course  on  a self-paced  basis  and  the  capabQity  to  review  lesson  material  at  any  time.  A third  area  of 
favorable  comments  uy  the  students  related  to  the  accessibility  and  concern  of  the  PLATO  staff.  During 
each  of  the  scheduled  sessions  at  least  one  monitor  was  avaOable  to  assist  students  with  either  technical  or 
subject-matter  problems.  If  the  monitor  was  unable  to  satisfactorily  answer  the  questions  related  to  subject 
matter,  the  author  of  the  lesson  in  question  was  immediately  avaQable  to  discuss  and  clarify  any  student’s 
questions  on  an  individual  basis.  Ihis  individualized  attention  was  an  additional  advantage  of  CAI  in 
contrast  to  the  lock-step,  large  classroom  lecture  approach  of  the  traditional  course  which  inhibited  an 
instructor’s  ability  to  pursue  individual  questions. 

The  retest  of  the  first  PLATO  class  coincided  with  the  early  termination  of  the  POMC  project  during 
the  second  trimester.  At  this  time,  eight  of  the  original  sixteen  students  (class  1)  were  in  the  program.  Five 
of  the  original  POMC  students  were  acaaemical  eliminated  which  was  similar  to  the  number  of  students 
(N=6)  eliminated  from  the  conventional  PA  cou:  ^e  during  the  same  time  period.  Three  students  remained  in 
Ae  PA  program,  but  withdrew  from  the  PLATO  project.  The  five  students  who  were  academically 
eliminated  were  interviewed  to  ascertain  if  their  involvement  in  the  PLATO  project  was  related  to  their 
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faflure  in  the  overall  course.  None  of  the  students  felt  that  there  was  any  correlation  between  their  failure 
and  the  PLATO  project.  Most  of  the  students  felt  that  the  traditional  course  was  too  rapid  and  found  the 
PLATO  CAl  programs  superior  in  comparison.  Two  of  the  students  felt  that  they  could  have  completed  the 
curriculum  if  the  entire  course  were  self-paced.  It  is  also  interesting  to  note  that  all  five  students  had 
completed  all  of  their  required  lessons  on  PLATO  and  all  had  a passing  average  in  the  PLATO  portion  of 
their  coursework. 

The  three  students  who  withdrew  were  also  interviewed.  One  student  felt  that  he  could  not  learn  as 
well  in  a self-paced  mode  as  he  could  from  a lecture.  He  stated  he  always  retained  more  iMien  he  could  hear 
the  information  and  then  take  notes.  Additionally,  he  felt  his  reading  comprehension  was  low  and  to 
continue  using  a medium  which  was  totally  visual  could  be  detrimental  to  his  success  in  the  overall 
program.  The  other  two  students  who  withdrew  felt  that  they  were  not  learning  as  much  as  their 
counterparts  in  the  traditional  classroom.  They  stated  that  the  PLATO  program  required  more  of  their 
time,  and  they  resented  the  fact  that  they  had  to  come  back  to  the  school  to  review  PLATO  CAI,  rather 
than  reviewing  notes  at  home  like  their  peers  in  the  lecture  group.  All  three  of  these  students  were  shown 
that  their  end-of-lesson  tests  and  overall  block  examinations  demonstrated  they  were  learning  well  and  were 
on  par,  or  ahead  of  the  students  in  the  traditional  course.  Despite  the  objective  test  evidence  and 
reassurance  of  the  PLATO  staff,  the  students  could  not  resolve  their  anxieties  about  being  involved  in  an 
irmovative  curriculum  unlike  their  former  mode  of  learning  via  conventional  lecture  methods. 

Of  the  eight  students  (class  1)  who  remained  in  the  project,  most  had  the  same  anxieties  about  not 
le?ming  as  mudi  in  the  POMC  PLATO  condition  as  in  the  traditional  classroom  lectures.  Based  on 
interviews,  the  primary  reason  for  student  concern  was  the  fact  that  there  was  not  an  hour-for-hour 
identical  replacement  of  material,  and  students  could  not  comprehend  why  their  counterparts  were  learning 
one  set  of  materials,  whereas  they  were  frequently  performing  entirely  different  learning  tasks.  The  concept 
that  the  POMC  was  drawing  the  required  information  into  a different  sequence,  but  all  the  material  would 
be'  covered  eventually  at  the  end  of  the  year,  never  appeared  to  be  adec'iate  to  resolve  student  concerns.  By 
the  end  of  the  second  trimester,  six  of  the  remaining  ei^t  students  requested  to  return  to  the  dassroom. 
Student  anxiety  about  not  learning  the  same  material  (and  consequently  perceiving  a high  risk  regarding 
successful  course  completion)  was  a major  contribution  to  the  termination  of  this  project.  A more  complete 
discussion  of  mis  problem  will  appear  later  in  the  integration  of  the  two  POMC  clases’  attitude  surveys. 

The  results  of  the  retest  of  the  fint  PLATO  class  indicated  their  primary  complaint  was  still 
frustration  with  system  malfunctions.  It  should  be  pointed  out  that  there  were  no  migor  system  crashes 
which  prevented  a class  from  using  the  system,  but  rather  the  consistent  disruption  of  their  work  by 
transmission  line  errors.  Line  errors  distort  the  readability  of  displayed  information.  The  next  area  of 
disfavor  was  the  need  for  acceptable  correct  answers.  In  theoretical . "rterial,  a concept  judging  sequence 
was  attached  to  the  answer  judger,  but  the  variety  of  student  answers,  though  correct,  could  never  be 
completely  anticipated.  Hence,  some  students  were  not  inunediately  credited  with  aconect  response.  The 
last  major  problem  area  was  that  the  PLATO  project  was  viewed  as  an  extra  course  and,  consequently,  the 
students  commented  on  the  extra  time  needed  to  do  PLATO  lessons.  This  misunderstanding  about  extra 
time  was  a second  major  contribution  to  the  decision  to  terminate  this  project,  and  also  reflects  rumor 
pervasiveness  and  distorti(!)n.  These  areas  will  be  elaborated  upon  in  a later  portion  of  the  discussion  of 
student  attitudes. 

On  the  attitude  survey,  none  of  the  test-retcst  nwan  item  scores  was  significantly  different  at  the  .10 
level  of  confidence.  However,  in  tl.e  task  of  deriving  meaning  from  small  samide  size  altitudinal  data, 
customary  statistical  analyses  may  be  less  appropriate  than  a contextual  analysb  of  information.  The 
following  analyses  were  conducted  by  the  PLATO  IV  staff  clinical  psychologist.  Attitude  items  for  vdiich 
there  was  a change  of  one-half  point  or  more  between  the  test-retest  means  were  sdected  by  the  PLATO 
staff  as  reflecting  a detectable  change  in  the  students’  attitudes  after  extended  exposure  to  PLATO.  These 
items  will  be  discussed  in  relation  to  student  iriterviews  conducted  by  the  staff  psychologist. 

Item  7 (Appendix  G)  indicated  on  the  retest  that  students  became  more  aware  that  their  answers 
were  correct  before  being  indicated  correct  on  the  screen.  Students  reported  that  as  they  became  more 
comfortable  with  the  terminal  and  more  experienced  with  the  lesson  riKthods,  they  became  more  confident 
of  their  responses.  An  increased  favorable  response  on  item  13  showed  that  stu^nts  felt  the  lessons  and 
computers  were  less  mechanical  as  their  exposure  to  PLATO  increased. 

Responses  to  items  21  and  22  appeared  to  be  in  contrast  to  the  comments  about  initial  test 
frustration  with  PLATO  «nd  inflexibility  of  acceptable  student  answen.  The  students  felt  that  the  lessons 


J 


20 


had  improved  such  that  students  progressed  more  easfly  through  the  lessons.  The  previously  mentioned 
system  malfunctions  continued  to  be  a problem,  but  the  improved  lessons  caused  a more  positive  attitude 
toward  computerized  education.  The  students  further  stated  that  the  improved  lessons,  experience  with 
PLATO,  and  better  understanding  of  the  lesson  material  made  it  easier  to  give  an  answer  acceptable  to  the 
computer.  It  appeared  that  the  students  had  developed  a system  by  which  they  could  quickly  determine 
what  answer  was  appropriate,  by  combining  the  information  from  the  hints  ^ven  with  a wrong  answer  and 
a quick  review  of  the  previous  lesson  material. 

The  improved  attitude  of  students  on  item  23  indicated  a more  favorable  attitude  toward  PLATO 
CAI  compared  to  initial  attitudes.  General  improvements  in  the  quality,  interaction,  and  animation  of  the 
coursework  made  the  POMC  material  more  interesting  and  meaningful  to  the  student.  Students  commented 
that  the  flist  series  of  courses  appeared  to  be  slow-moving  and  boring,  but  the  second  trimester  courses 
were  more  enjoyable  and  educational. 

Item  27  indicated  that  CAI  was  not  viewed  as  a depersonalizing  experience,  perhaps  because  at  least 
one  monitor  was  available  at  aH  times  and  any  problems  could  be  solved  immediately  on  a one  to-one  basis. 
Students  reported  that  although  they  were  working  independently,  me  knowledge  that  personalized 
assistance  was  available  removed  any  depersonalizing  aspect  from  the  learning  situation. 

On  item  10,  the  students  were  not  interested  in  finding  out  more  about  the  subject  matter  because 
they  felt  that  the  major  points  had  been  adequately  presented.  They  did,  however,  desire  handouts  or 
reprints  to  carry  with  them  for  review  purposes. 

Items  11  and  19  were  reiterations  of  the  same  point.  Students  felt  (item  II)  that  they  were 
over-involved  with  running  the  terminal  because  of  the  consistent  disruption  of  their  woil:  due  to  line 
errors.  Item  19  reflects  student  annoyance  when  answers  were  not  accepted  and  at  other  times,  when  part 
of  the  text  would  not  appear,  or  there  would  be  serious  overprinting.  This  required  extra  steps  on  the 
student’s  part  to  retype  the  answer  or  go  back  in  the  lesson  to  get  a readable  reprint  of  the  text  which  was 
missed.  I^oblems  of  this  nature  were  the  difficulties  constantly  discussed  by  the  students  as  being  a 
frustration  with  PLATO  IV. 

Class  II.  The  second  PLATO  class  to  experience  first  trimester  material  was  administered  the  initial 
attitude  survey  after  six  weeks  of  POMC  material  (approximately  seven  hours  of  coursework).  At  that  time, 
the  students  reported  considerable  problems  with  system  mdfunctions  and  difficulty  with  acceptable 
answen  being  too  specific.  Based  upon  feedback  and  problems  with  lessons  which  were  encountered  by  the 
first  PLATO  class,  many  modifications  and  improvements  had  now  been  made  to  the  first  trimester 
material.  The  improvements  in  lesson  sequencing  were  successful  and  eliminated  any  comment  by  the 
studenb  concerning  difficulties  progressing  through  the  lessons.  In  addition  to  the  sequencing  modification, 
most  of  the  “answer  judging  concepts"  were  expanded  to  accept  a wider  range  of  correct  responses; 
however,  the  expanded  range  still  did  not  succeed  in  anticipating  the  combination  of  words  given  by  the 
second  dass,  as  documented  by  the  comments  on  acceptable  answers  being  too  specific.  Improvements  in 
lesson  material  were  reflected  in  favorable  student  comments  about  course  material  in  the  second  POMC 
class.  Malfunctions  which  students  found  irritating  were  not  major  system  crashes,  but  rather  the  frequent 
momentary  disruptions  resulting  from  line  errors.  The  next  major  area  of  discontent  was  student  perception 
of  the  POMC  project  as  an  “extra  course.”  Despite  numerous  attempts  by  the  PLATO  staff  to  inform 
students  that  the  POMC  material  was  a replacement  of  class  material,  it  was  never  fully  accepted  by  the 
students.  POMC  student  concern  about  what  their  counterparts  were  learning  and  the  fact  that  the  PI  ATO 
material  was  not  identical  to  the  lecture  material  caused  considerable  anxiety  about  participating  in  the 
project. 

There  were  mixed  feelings  about  the  reriew  capability  in  the  first  trimester.  Three  students  felt  the 
review  capability  was  a strong  positive  feature  of  the  PLATO  program,  while  five  students  felt  it  took  too 
much  time  to  review.  This  acute  awareness  of  time  factors  reflected  the  pressures  that  these  students  felt 
about  their  time.  Delay  or  disruption  of  their  schedule  was  regarded  a major  difficulty.  Another  favorable 
comment  described  the  staff  as  being  interested  in  the  student’s  progress  and  staff  accessibility  as  being 
helpful  to  the  students.  The  final  major  favorable  comment  was  that  PLATO  provided  variety  to  the 
standard  curriculum. 

The  comments  on  the  retest  of  attitudes  indicated  that  the  pressures  of  time  caused  concern  that 
partfdpation  in  PLATO  hindered  success  in  the  overall  course.  Frustration  with  system  malfunctions  also 
remained  a prominent  complaint  of  the  students.  The  irritation  with  the  specificity  of  acceptable  answen 
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diminished  somewhat  as  the  students  became  more  accustomed  o this  innovative  teaching  system.  The 
favorable  comments  on  the  retest  still  listed  self-pacing,  review  capability,  and  attitude  of  the  PLATO  staR’ 
toward  the  students  as  the  three  major  benefits  of  the  PLATO  experience. 

No  statistically  significant  differences  (p  < .10)  were  found  between  test-retest  scores  in  dass  II. 
Customary  ^.tatistical  analyses  wcie  deemed  by  the  PLATO  staff  to  be  less  meaningful  than  contextual 
interpretation  by  the  staff  cljiical  psychologist.  Only  items  which  had  a change  of  one-half  point  or  more 
between  the  initial  scale  and  the  fstest  were  regarded  as  a qualitative  index  of  change. 

The  most  noticeable  improvement  in  student  attitude  occurred  in  the  reduction  of  feelings  about 
isolation  (item  4)  and  depersonalization  (item  27)  in  computerized  instruction.  As  the  students  had  more 
exposure  to  PLATO,  they  became  more  contfortable  with  working  on  their  own.  This  was  a marked 
accomidishment  since  8C%  of  the  students’  time  was  spent  in  large  classes  and  study  groups  for  afterdiours 
study  sessions. 

Students’  anxieties  about  not  learning  the  same  material  as  the  control  group  and  pressures  of  time 
generated  many  negative  feelings  about  PLATO  coutsework.  Students  questioned  the  productivity  of  the 
time  spent  on  PLATO  (items  20  and  24).  The  notion  of  working  at  their  own  pace,  which  was  a positive 
aspect  in  initial  comments,  later  precipitated  problems.  The  students  felt  they  could  work  at  their  own 
pace,  but  found  that  this  did  or  could  present  problems  if  they  exceeded  PLATO  classroom  hours.  If 
students  exceeded  PLATO  hours,  important  information  in  the  on-going  conventional  dassroom  might  be 
missed.  Likewise,  PLATO  students  not  completing  PLATO  lessons  within  the  scheduled  time  periods 
produced  aii  additional  tradeoff.  Additional  time  had  to  be  rescheduled  after  the  regular  PLATO  class, 
which  was  an  infringement  on  study  time  for  assignments  in  the  conventional  dass.  Thus,  if  students 
worked  at  their  own  pace,  it  was  . welcomed  change,  but  it  became  problematic  when  they  tried  to 
schedule  extra  time.  The  students  further  commented  that  the  terminal  was  too  slo\>.  They  felt  that  the 
text  took  too  tong  to  appear,  graphics  took  too  much  time  to  complete,  and  animations  were  sometimes 
distracting.  When  interviewed  fur^er  about  these  attitudes,  students  repeated  that  they  did  not  have  time 
to  watch  the  animations,  work  the  graphs,  or  wait  for  text.  This  led  to  boredom  (item  30)  and  guessing  at 
answers  (item  8)  in  order  to  get  the  hint  from  the  -HELP-  sequence  which  would  allow  them  to  speed  up 
their  progress  through  a lesson. 

Integrated  Attitudinal  Data  Discussion.  Review  of  overall  attitudinal  data  disclosed  several  areas 
important  to  an  understanding  of  problems  encountered  in  the  POMC  project.  Additionally,  the  following 
information  constitutes  important  lessons  to  be  learned  for  development  of  future  computer-based 
education  (CBE)  programs  in  the  military  environment. 

The  first  area  to  be  explored  was  an  observable  change  in  student  attitudes  with  exposure  to  CBE. 
Initially,  both  the  first  and  second  classes  were  apprehensive  about  engaging  in  an  innovative  learning 
program.  Concerns  about  both  the  value  of  the  coursework  and  the  capability  of  the  software  were 
prevalent.  The  new  PLATO  IV  medium  appeared  to  be  a source  of  frustration.  As  students  worked  their 
way  through  lessons,  they  were  perplexed  by  the  use  of  various  function  keys,  such  as  -HELP—  and  BACK. 
This  was  reflected  in  their  comments  about  distractions  by  the  terminal.  As  *heir  experience  and  expertise 
using  the  keyboard  increased,  this  frustration  diminished  and  there  was  a noticeable  shift  in  the  students’ 
ttention  from  the  mechanics  of  the  operation  to  the  educational  value  of  the  material. 

A second  area  of  common  concern  was  the  students’  apprehensions  about  not  learning  as  mudi  as 
their  counterparts  in  the  traditional  lecture.  Despite  numerous  supportive  counseling  sessions  by  the 
PLATO  staff  and  the  introduction  of  objective  test  results  which  cleaiiy  demonstrated  successful  leamirtg 
of  course  objectives,  students  persisted  in  the  misconception  that  they  were  falling  behind  dieir 
counterparts  in  the  traditional  Lcture  system.  Attempts  to  dispel  this  concern  were  ccmiplicated  by  the  fact 
that  the  POMC  was  an  integrated  curriculum  drawing  information  from  several  areas  encompassing  subject 
matter  from  all  three  trimesters  rather  than  an  hour-for-hour  subject-matter  replawment.  Students  could 
not  be  convinced  that  the  POMC  program  would  eventually  cover  all  of  the  required  material,  because  the 
sequence  was  not  the  same  as  depicted  in  the  traditional  course  outline.  Moreover,  the  instructional  topics 
and  sequence  differed  from  their  counterparts  in  the  lecture  condition.  The  staff’s  inability  to  resolve  the 
students’  concerns  about  the  POMC  and  PLATO  project  was  further  complicated  by  the  fact  that 
approximately  90  percent  of  student  time  was  spent  in  a rapid  group-paced,  lock-step  curriculum. 
Transistion  to  this  self-paced,  innovative  approach  to  learning  created  apprehension  rather  than  alleviation 
of  pressures  related  to  learning. 
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lune  constraints  and  voluminous  amounts  of  material  to  be  learned  added  a third  dimension  of 
complicatiim  to  the  students’  attitudes  and  psychological  acceptance  of  the  project.  As  previously 
mentioned  in  the  introductory  description  of  the  PA  student’s  curriculum,  the  student  engaged  in  eight 
hours  of  continuous  class  work  each  day  which  was  followed  by  three  to  six  hours  of  study  each  ni^t. 
There  wa:  no  latitude  for  digression  from  this  schedule  and  any  unanticipated  addition  to  their  learning 
requirements  had  many  adverse  affects.  The  students’  misconceptions  about  the  POMC  created  the 
distorted  opinion  that  the  PLATO  project  was  an  “extra  course.”  This  view  of  the  project  resulted  in  the 
POMC  students  feeling  that  they  were  in  a way  penalized  by  being  involved  in  PLATO. 

This  combination  of  fear  of  academically  falling  behind  their  peers,  misconceptions  about  the  POMC, 
and  viewing  the  PLATO  project  as  an  “extra  course”  produced  a severe  attitude  problem  which  could  not 
be  resolved.  The  students’  negative  attitudes  were  reinforced  by  the  see-saw  alternation  between  rapid 
paced  lehures  and  self-paced  instruction,  from  large  lecture  classes  to  CAI  tutorial  instruction,  and  from 
traditional  didactic  information  to  an  innovative,  integrated  curriculum.  With  all  of  the  external  pressures 
of  the  overall  curriculum  and  the  realization  that  their  military  careers  could  be  affected  by  their 
performance  in  the  PA  course,  these  attitudes  compelled  POMC  students  to  request  to  be  removed  from  the 
PLATO  project.  It  was  at  this  point  that  the  Sheppard  PLATO  staff  decided  to  terminate  the  POMC  project 
because  nothing  further  could  be  done  to  resolve  the  misconceptions,  relieve  the  anxieties,  or  change  the 
negative  attitudes  of  the  students.  Furthermore,  continuation  of  the  project  could  be  detrimental  to  the 
students’  overall  learning  and  lead  to  erroneous  interpretation  of  subsequent  research  data. 


Vn.  STUDENT  PERFORMANCE 

Originally,  the  primary  comparison  of  academic  performance  anticipated  to  have  the  most  validity, 
was  an  end-of-course  board  certification-type  examination  to  be  develo(>ed  from  a pool  of  questions  from 
civilian  PA  programs.  This  examination  would  have  permitted  a comprehensive  examination  of  both  groups’ 
knowledge  of  the  respiratory  system  and  related  diseases  after  all  couisework  was  completed.  Rationale  for 
this  examination  as  a comparative  measure  of  academic  performance  included  the  notion  that  regardless  of 
whether  a student  experienced  the  POMC  or  the  traditional  lecture  system,  both  groups  mi^t  be  expected 
to  have  a similar  amount  of  basic  knowledge  of  respiratory  systems  and  respiratory  illnesses  upon 
completion  of  1400  hou.s  of  training  at  the  end  of  the  first  year.  However,  since  less  than  10  percent  of  the 
total  curriculum  could  not  be  expected  to  produce  achievement  differences  on  the  comprehensive  end  of 
year  board  examination,  statistically  significant  achievement  differences  between  the  POMC  and  traditional 
lecture  were  improbable.  In  addition,  thou^  a valid  comparison  was  theoretically  possible,  dwindling 
sample  size,  student  reactions  and  differential  confounding  factors  discussed  previously  precluded  valid 
interpretative  comparisons.  The  discussion  of  performance  data,  which  follows,  is  therefore  presented  as  an 
attempt  by  the  Sheppard  PLATO  staff  to  ^are  their  experiences  and  lessons  learned  with  colleagues 
contemplating  the  use  of  computers  in  an  operational  environment. 

It  was  expected  that  intermediate  comparisons  of  POMC  end-of-lesson  and  'olock  tests  could  not  be 
compared  directly  with  end-of-lesson  or  block  examinations  in  the  traditional  system  because  of  differences 
in  sequencing,  presentation  of  material  and  test  items.  Since  these  performance  measures  could  not  l<e 
compared,  evaluation  of  remaining  comparative  measures  of  student  performance  was  attempted  by  means 
of  logical  analyses  and  personal  knowledge  of  the  conditions.  Despite  these  obrious  limitations,  there  wetw 
some  useful  comparisons  which  wfll  be  discussed  in  the  context  of  the  above-mentioned  constraints. 

The  first  area  of  performance  evaluation  to  be  discussed  is  that  of  student  time  spent  on  POMC 
lessons  compared  v'Uh  actual  number  of  hours  replaced  in  the  traditional  course.  In  the  first  trimester,  the 
"POMC  replaced  14  classroom  hours.  The  actual  time  spent  on  the  computer  by  the  POMC  students  was  15 
hours  53  minute.s,  as  collected  by  the  lesson  timing  program  developed  by  the  SHCS  PLATO  staff.  Tliis 
method  of  timing  was  an  improvement  over  the  old  method  because  it  calculated  actual  time  in  the  course 
without  adding  timing  artifacts.  Thus,  15  hours  and  53  minutes  included  review  time.  The  actual  time  to 
initially  complete  the  course  material  was  1 1 hours  27  minuteis;  therefore,  4 hours  and  26  minutes  were 
spent  in  review.  If  the  ATC  developed  review  time  requiiement|  is  added  to  the  traditional  hours,  14  hours 
of  coursework  and  4.5  hours  of  study  or  review,  sums  to  a total  of  18.5  hours.  Using  these  figures,  the 
POMC  students  used  approximately  2.5  houn  less  of  overall  classroom  plus  study  time.  A statistical 
comparison  was  not  warranted  because  there  was  no  way  to  determine  exactly  how  much  study  time  the 
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traditional  course  students  used  for  review.  In  addition,  some  POMC  students  did  review  some  of  the 
material  off-line;  this  review  time,  of  course,  was  not  recorded.  Whatever  addition  or  subtraction  of  the 
unknown  study  hours,  it  was  very  doubtful  that  there  were  any  significant  time  differences  between  the 
POMC  and  lecture. 

There  is  scxne  validity  to  speculating  that  the  POMC  students  in  the  first  trimester  required  less  time 
for  initial  study  and  review.  This  fairly  objective  finding  about  time  spent  in  POMC  was  particularly 
significant  when  compared  with  the  attitudini  data  previously  discussed.  TTie  POMC  students’  feelings  that 
they  had  an  extra  course  which  demanded  more  time  was  definitely  an  attitudinal  discrepancy  rather  than  a 
reality.  Further  comparisons  of  time  could  not  be  clearly  developed  because  the  fust  trimester  times  for  the 
first  test  group  used  a less  sensitive  and  less  precise  timing  recorder,  and  the  second  trimester  times  v/ere 
incomplete  because  several  students  had  not  completed  all  of  the  course  material.  The  best  summary  of 
time  canparison  data  appears  to  be;  (a)  based  upon  the  limited  sample  of  data,  it  appeared  that  the  POMC 
students  were  not  required  to  >'xpend  any  extra  time  in  clas-sroom  work  and  study;  and  (b)  perhaps  there 
was  a slight  reduction  in'overail  time  when  compared  with  the  traditional  classroom  hours  and  study  time. 

Summary  statements  regarding  student  performance  must  be  made  in  a very  general  context  due  to 
the  small  sample  sizes  and  differing  criterion  measures.  A between-group  (FOMC-Lectv.re)  acadetric 
comparison  was  not  warranted  due  to  the  differences  in  subject-matter  content,  sequence  and  ciiterion 
measures.  However,  a within-group  assessment  of  POMC  instructional  effectiveness  on  POMC  students 
solely  may  provide  some  estimate  of  educational  effectiveness. 

POMC  students’  mean  score  (77.6,  SD  6.90,  N=8)  on  a block  area  iCst  over  specific  instructional 
objectives  taught  via  PLATO  did  not  differ  significantly  from  the  same  POMC  students’  scores  (79.0,  SD 
4.14,  ?'=8)  on  other  first  trimester  instructional  objectives  presented  by  the  lecture  method.  Tliough 
achievement  performance  increased  for  the  second  class  of  students  who  experienced  both  POMC  and 
lecture  methods  on  different  instructional  objectives,  the  achievement  differences  were  non-significant 
(POMC  90.0,  SD  5.53;  lecture  84.6,  SD  8.18).  It  mi^t  be  argued  that  the  POMC  material  was  at  least  as 
effective  in  teaching  material  in  the  FA  course  as  the  traditional  lecture  and  that  this  instructional 
effectiveness  was  achieved  with  possibly  some  overall  student  time  savings. 

One  other  important  factor  which  influenced  student  academic  performance  was  the  temporal 
sequencing  of  the  curriculum.  In  the  first  trimester,  13  hours  of  class  material  was  spread  over  three 
monthi  The  students  received  nine  hours  of  class  material  over  a one-month  period  and  the  tenth  hour  did 
not  occur  until  eight  weeks  later.  Although  the  basic  material  may  ~ it  have  been  continuous  in  nature,  the 
basic  POMC  problen.-solving  orientation  certainly  was  not  facilitated  by  the  lengthy  disruption  in 
presentation. 

The  second  trimester  demonstrated  the  same  scheduling  problem.  It  was  explained  that  the  POMC 
was  a method  of  training  students  to  incorporate  basic  science  material  into  a clinical  problem-solving 
mode.  This  learning  process  was  difficult  enou^  in  an  intense,  concentrated  teaching  approach,  but  was 
virtually  impossible  using  the  scheduling  sequence  which  was  dictated  by  the  implementation  agreement. 

The  preceding  findings  are  very  general  and  tenuous  because  of  the  numerous  constraints  on  the 
potential  for  statistical  analysis  and  direct  comparisons.  To  avoid  these  problems,  future  research  should  be 
designed  to:  (a)  Permit  block-for-block  materiab  replacement  on  an  hour-for-hour  schedule^  This  would 
allow  direct  comparison  of  results  with  minimal  complication  of  conUmination.  (b)  Avoid  the  pitfalls-  cf 
presenting  small  amounts  of  material  over  a prolonged  period  of  time,  (c)  Provide  ample  time  for  lesson 
completion.  Courses  selected  should  be  planned  to  allov'  students  to  take  the  entire  lesson  within  a short 
period  of  time  to  enable  students  to  maximize  transfer  of  information,  the  benefits  of  reinforcement,  and 
drill  and  practice,  (d)  Select  courses  which  have  a high  student  flow  to  insure  an  adequate  sample  size,  (e) 
^quire  cleariy  defined  criterion  objectives  and  measurement  instruments  used  to  evaluate  dimensions  of 
instructional  effectiveness;  e.g.,  degree  to  which  newly  acquired  knowledge  and  skills  are  retained  and 
correctly  applied  to  problems  representative  of'those  most  likely  to  be  encountered  on  the  job  in  medical 
occupations. 

One  final  area  of  pre-planning  for  future  research  concerns  the  establishmeiu  of  both  pre-  and 
post-measures  of  student  characteristics  to  include  pre-course  knowledge  of  subject  matter.  The 
inrarporation  of  these  design  features  would  serve  to  specify  the  conditions  contributing  to  instructi'-nal 
effectiveness  and  would  assist  in  data  interpretation. 
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VIO.  C»ST  ISSUES/RECOMMENDATIONS 


The  purpose  of  tWs  section  is  to  provide  data  on  the  following  cost  areas;  (a)  Setting  up  an 
organization,  (b)  Procuring  PLATO  equipment  required  to  support  computer-based  lessons,  (c)  Preparing 
existing  classroom  facQities  for  PLATO  implementation,  (d)  Training  PLATO  programmers  and  techmcd 
support  pei.;onnel.  (e)  Recurring  coirputer  support  costs.  (0  Persotmel  costs,  (g)  Support  costs. 

In  addition  to  these  cost  ireas,  a number  of  factors  which  could  significantly  affect  the  cost  of  future 
computer-based  education  projects  wfll  be  discussed.  The  purpose  of  this  latter  cost  examination  is  to 
insure  that  consideration  is  given  to  a number  of  significant  variables  prior  to  finalizing  any  plan  for  the 
operational  implementation  of  a PLATO-Iike  computer  educational  system. 

Hardware  Costs.  The  PLATO  IV  hardware,  located  within  the  School  of  Health  Care  Sciences,  was 
procured  by  .ARPA  through  contract  with  the  University  of  Illinois'  Center  for  Education  Research 
Laboratory  (CERL),  IniMal  hardware  investment  costs  are  reflected  in  Table  3. 


Table  3.  Initial  Hardware  Investment  Costs 


lt*m 

Quantity 

Total  Cott 

Basic  Terminal  and  Keyset  ($5,000  ea)* 

20 

100,000 

Slide  Selectors  ($600  ea) 

20 

12,000 

Touch  Panels  ($410  ea) 

20 

8,200 

Multiplexers  ($4,680  ea) 

5 

23,400 

Connent  (Port)  Cost  ($300  ea) 

5 

1,500 

Audio  Unit  ($2,000  ea)** 

2 

4,000 

$149,100 

•Basic  terminal  and  keyset  prices  nuctuated  between  $4,805  and  $5,215  each. 
The  average  price  for  each  SHCS  terminal  amounted  to  $5,000. 

following  Initial  experimentation,  the  random  access  audio  devices  were 
returned  to  CERL  and  not  used  in  the  SHCS  research  effort. 


Set-Up  Costs.  To  support  the  PLATO  research  staff  and  twenty  PLATO  IV  terminals  with  associated 
communications  equipment,  three  30-  by  30-foot  rooms  were  made  available  within  the  School  of  Health 
Care  Sciences  (budding  1900)  at  Sheppard  Air  Force  Base,  Texas.  Setup  costs,  induding  equipment, 
supplies  and  budding  modifications,  are  itemized  in  Table  4. 

Total  setup  costs  amounted  to  SI  8^41.74.  It  diould  be  noted  that  building  19(X>  possessed  a 
compressed  air  supply  and  sufficient  power  and  air  conditioning  to  support  the  PLATO  hardware.  The  only 
setup  costs  incurred  in  these  areas  resulted  from  tapping  into  the  existing  systems. 

Programmer  and  Technical  Support  1>aining.  All  SHCS  PLATO  pereonnel  except  the  medical 
illustrator  and  stenographic  spedalist  attended  a five-day  training  course  at  the  University  of  Illinois.  The 
course  included  the  use  of  the  TUTOR  programming  language  and  lesson  development  protocol  for  CBE 
lessons,  ^ollow-on  training  was  conducted  in-house  by  visiting  University  of  Illinois  representatives  and 
members  of  the  SHCS/PLATO IV  technical  support  section. 


Round  trip  Air  Fare  (SHCS  to  U of  I) 
Per  Diem  S25/day  (5  days) 

S321.22X  11  =$3,533.42 


$196.22 

125.00 

$321.22 
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Table  4.  Setup  Costs 


Equipment 

Quanllly 

Total  COit 

Drawing  Table 

1 

439.00 

Kodak  RA950  Projector 

2 

1,184.00 

Slide  Display/^iew  Cabinet 

1 

409.86 

Davenport 

1 

134.00 

Lounge  Chair 

2 

108.00 

Metal  Office  Table 

3 

264.00 

Desk 

13 

2340.00 

Wood  Office  Table 

2 

142.00 

Desk  (Typing) 

1 

103.00 

Storage  Cabinet 

2 

200.00 

Programming  Board 

1 

71.00 

Selectric  Typewriter 

1 

601.88 

Manual  Typewriter 

1 

197.25 

Magnetic  I^corder 

1 

409.97 

Magnetic  Reproducer 

1 

460.10 

Vacuui.i  Qeaner 
Supplies: 

1 

186.00 

$7,250.06 

Office  Partitions;  Accessories: 
Building  Modifications: 

$2,060.62 

Electrical  Power  and  Compressed  Air 

$2,821.06 

Student  Carrels  (20  x $260.50) 

5,210.00 

Carpeting  (100  sq  yds  @ $10  sq  yd) 

1,000.00 

$9,031.06 

Recurring  Computer  Support  Costs.  Recurring  computer  support  costs  were  allocated  on  a per 
terminal  basis  for  computer  support  and  maintenance  and  on  a per  mile  basis  (700  miles)  for 
communications  lines.  Four  terminus  and  one  communication  line  were  operational  in  March  1974,  while 
the  remaining  16  terminals  and  4 communications  lines  became  operational  during  October  1974.  Termmal 
support  w *s  $2,500  per  year  per  terminal.  This  figure  includes  $ 1 ,500  for  computer  support  and  $ 1 ,000  for 
maintena  ;ce  to  include  parts.  Communications  costs  per  line  according  to  ARPA  was  $427.35  per  month 
resulting  in  $25,641  per  year  for  five  GSA  lines.  Total  recurring  computer  support  costs  are  reflected  as 
follows: 


1974 

4 terminals  X (10  months  X $2,500) 
1 line  X (10  months  X $427.35) 

16  terminals  x (3  months  x $2,500) 
4 lines  x (3  months  x $427.35) 


= $ 8333J0 

= 4,273.50 

= 10,000.00 

= 5,128.20 

$27,735.00 


1975 

20  terminals  X (12  months  X $2,500)  = $50,000.00 

5 lines  X (12  months  X $427.35)  = 25,641.00 

$75,641.00 

1976  (through  1 March  1976) 

20  terminals  x (2  months  x $2,500)  = $ 8,333.30 

5 lines  X (2  months  X $427.35)  = 4*273.50 

$12,606.80  ’ 

..  recurring  computer  support  costs  for  the  SHCS  POMC  effort  from  March  1974  to 

March  1976  amounted  to  $1 15,982.80. 
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Personnel  and  Support  Costs,  The  composition  of  the  SHCS  PLATO  IV  Development  Branch  is  listed 
elsewhere  in  this  report  (Appendix  H).  The  personnel  costs  were  extracted  from  standard  Air  Force  cost 
reporting  documents  and  do  not  include  overhead.  Consequently,  actual  costs  may  have  varied  slightly  from 
the  flgutes  shown  as  follows: 


FY74  (January  1975-June  1975) 


Military  Pay 

20,504 

Civilian  Pay 

2,753 

Supplies  (excludes 

GSA  partitions) 

589 

$ 23,846 

FY75 

Military  Pay 

216,788 

Qvilian  Pay 

9,543 

TDY 

1,500 

Supplies 

1,177 

$229,008 

FY76 

Military  Pay 

161,556 

Civilian  Pay 

6,936 

TDY 

1,456 

Supplies 

809 

$170,757 

Total  penonnel  and  support  costs  (excluding  GSA  ofHce  partitions  and  equipment  costs  which  are 
reflected  elsewhere)  for  the  period  1 January  1974  to  1 March  1976  amounted  to  S423,61 1. 

Toted  Cost  Summary.  It  Aould  be  noted  that  the  costs  reflected  herein  do  not  include  ARPA 
contract  costs  for  CERL  services,  ($206,000)  which  included  lesson  reviews  and  critiques,  programming 
assistance,  hardware  development,  nor  staff  assistance  visits  to  SHCS  by  CERL  representatives.  Also 
excluded  are  the  costs  of  stafl'  assistance  visits  performed  by  ATC/SG,  ATC/XP,  and  AFHRL/TT 
representatives.  In  summary,  total  costs  by  category  for  the  period  1 January  1974  to  1 March  1976  are 
reflected  as  follows: 

Set-Up  Costs  $ 18,441.74  (SHCS) 

Hardware  Costs  149,100.00  (ART  A) 

IVogrammer  and  Te'-bnical  Support  Training  3,533.42  (SHCS) 

Recurring  Computer  Support  1 15,929.00  (ARPA) 

Personnel  and  Support  Costs  423,61 1.00  (SHCS) 

$710,615.16 

Operating  costi  for  the  SHCS  amounted  to  $445,486.16.  Addition  of  ARPA  support  in  the  hardware 
and  recuning  computer  support  areas  amounted  to  $265,082.80  which  when  summed  with  operating  costs 
yields  a total  cost  of  $710,568.96. 

Hypothetical  Operational  Costs.  The  cost  of  two  hypothetical  “operational”  PLATO  organizations  is 
depicted  as  follows;  both  proposed  units  would  be  located  under  the  SHCS  Training  Services  Division 
(MST).  The  concept  behind  organization  A is  that  the  uniqueness  of  the  PLATO  system  warrants  the 
establishment  of  the  PLATO  unit  as  a separate  entity  under  MST.  It  would  have  the  same  organizational 
and  operational  level  as  that  enjoyed  currently  by  the  Learning  Resource  Center  (LRC). 

Orianization  “A" 


Grade 

AFSC 

Title/Position 

Annual  Salary* 

I Capt 

9XXX 

Chief 

$20,751 

ITSG 

751X3 

Instructional  Systems  Tech 

12,336 

1 TSG 

9XXXX 

Department  Liaison 

12,336 

2SSG 

511X1 

TUTOR  Programmer 

18,062 

The  yearly  cost  of  this  organization  is  projected  at  $63,485. 

'Annual  salary  baKd  on  military  standard  rates  effective  I October  1975 
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Organization  B is  based  on  the  premise  that  the  following  currently  exists  within  the  LRC:  the 
management  philosophy,  and  expertise  and  flexibility  required  to  integrate  a medium  possessing  the 
capability  of  PLATO  into  the  current  organizational  structure. 

Organization  B 


Grade 

AFSC 

Title/Position 

Aimual  Salary* 

1 MSG 

9XXXX 

Supervisor/Department  Liaison 

S 14360 

ITSG 

751X3 

Instructional  Systems  Tech 

12,336 

2SSG 

511X1 

TUTOR  Programmer 

18,062 

The  projected  yearly  cost  of  this  organization  is  $44,758. 

^Annual  salary  based  on  military  standard  rates  effective  1 October  1975. 

Cost  Considerations.  It  is  readily  apparent  that  the  implementation  of  a CAI  program  is  an  expensive 
proposition.  However,  if  instructional  time  savings  and  the  quality  of  student  graduates  are  demonstrated  to 
be  superior  to  conventional  instruction,  an  examination  of  the  various  cost  components  involved  indicate 
that  a number  of  alternatives  exist  which  could  significantly  reduce  the  overall  expenditure  level  for  an 
effort  of  this  type. 

For  example,  persotinel  costs,  which  exceeded  SO  percent  of  the  project’s  total  costs,  are  a function 
of  the  number  of  individuals  assigned  and  their  relative  grade  level.  Significant  operational  savings  cuuld  be 
reaiized  if  consideration  were  given  to  converting  a number  of  officer  grade  author  positions  to  enlisted 
grade  SI  1X1  Computer  Programmers.  These  programmers  would  then  form  the  nucleus  of  lessor 
development  teams  which  would  indude  one  or  more  subject-matter  specialists  temporarily  assigned  (89 
days)  for  the  purpose  of  developing  individual  lessons.  Two  weeks  of  intensive  in-house  instruction  in  the 
areas  of  TUTOR  programming  and  instructional  materials  development  would  insure  a working  knowledge 
of  computer-based  lesson  development  among  the  subject-matter  specialists.  The  rank  of  these 
subject-matter  specialists  would  be  dictated  by  the  nature  of  the  subject  matter  being  developed.  The 
impact  of  this  “small  permanent  cadre-augmented  team”  approach  is  a significant  cost  reduction  (based  on 
numbers  and  grade  levels)  coupled  with  an  influx  of  fresh  instructional  ideas  and  approaches. 

Another  major  area,  that  of  recurr’tg  computer  support,  also  shows  a potential  for  cost  savings. 
Maintenance  data  should  be  accumulated  and  andyzed  to  determine  whether  terminal  maintenance  and 
spares  should  be  contracted  for  on  a yearly  basis  (currently  Sl,000/terminal/year)  or  funded  on  an  as 
required  basis. 

Besides  these  specific  examples,  the  following  factors  will  greatly  influence  the  operational  costs  of 
computer-based  education.  As  the  number  of  available  lesson  hours  increases,  the  number  of  student 
contact  houn  will  increase.  As  student  contact  hours  increase,  operational  costs  should  remain  constant,  up 
to  the  limit  allowed  by  the  configuration  of  the  system,  and  the  cost  per  student  will  decrease. 
Additionally,  the  number  of  lessons  developed  has  a unique  impact  on  the  cost  picture.  As  additional 
lessons  are  developed,  costs  increase  (additional  use  of  sulqect-matter  specialists  and  programmers)  but  can 
be  offset  by  increased  student  contact  hours  as  indicated  before.  However,  once  the  course  material  is 
developed,  the  only  programming  required  is  that  required  to  remove  any  errors  found  and  to  update 
materials.  Actual  programming  costs  drop  once  cour^  are  established. 

Location  of  student  terminals  can  also  directly  affect  cost.  Decentralization  of  student  terminals 
makes  the  terminals  readily  available  to  the  students;  however,  personnel  costs  are  increased  by  the 
muirement  for  additional  monitors.  This  depends  upon  the  necessity  of  having  a monitor  in  the  room. 
Centralized  terminal  location  minimizes  monitor  costs,  yet  reduces  ease  of  use. 

The  number  of  hours  the  terminals  ate  available  for  student  use  also  requires  examination.  The 
number  of  student  conuct  hours  increases  with  expanded  operational  hours;  however,  expanded  hours 
require  increased  monitoring  costs.  Therefore,  it  is  essential  that  an  opfimal  operational  time  be  established 
to  insure  the  most  cost-effective  use  of  existing  resources.  One  must  also  consider  the  cost  of  developing 
supportive  audio  visuals  and  other  supplementary  materials  when  planning  for  the  r^erational 
implementation  of  computer-based  education. 

Experience  is  an  extremely  important  factor  in  the  area  of  programming.  Data  gathered  from  many 
CBE  research  efforts  indicate  that  development  {lesson  programming)  time  decreases  with  experience. 
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Therefore,  it  is  essential  that  a small,  well-trained  cadre  of  experienced  personnel  be  maintained  pending  a 
decision  to  operationally  implement  CBE  on  a large  scale  in  the  medical  military  training  environment. 


IX.  SUMMARY  AND  RECOMMENDATIONS 

The'POMC  PLATO  project  was  originally  conceived  as  a research  project  to  compare  POMC  taught  by 
a computer  to  POMC  taught  in  a traditional  lecture  mode.  As  previously  explained,  modification  of  the 
traditional  lecture  course  to  accommodate  a POMC  format  did  not  occur.  Comparison  of  POMC  PLATO 
CAl  lessons  with  the  traditional  lecture  system  was  deemed  in  1974  to  be  a viable  research  project  and  was 
attempted.  Although  the  project  was  terminated  due  to  constraints  which  confounded  valid  evaluation, 
many  valuable  lessons  and  considerable  information  about  CAl  potential  in  Air  Force  medical  training  were 
obtained.  This  information  is  presently  being  employed  in  segments  of  the  Medical  Laboratory,  Radiology, 
and  Dental  Assistant  courses  at  Sheppard  AFB,  Texas. 

The  PLATO  staff  and  many  of  the  instructors  of  the  traditional  PA  program  felt  strongly  that  a 
problem-oriented  curriculum  had  many  advantages  and  could  be  the  most  advantageous  wav  to  teach 
medical  material.  The  potential  instructional  value  of  the  material  developed  by  the  PLATO  staff  was  seen 
as  having  considerable  usefulness.  The  consensus  of  opinion  was  that  a POMC  does  have  potential  in  Air 
Force  medical  education. 

Tire  use  of  CAl  in  the  military  medical  training  environment  was  also  seen  to  have  potential.  From 
staff  experiences  with  PA  training,  several  important  lessons  were  learned  regarding  selection  of  areas  to  be 
taught,  implementation  safeguards,  and  evaluation  techniques.  For  example,  lessons  considered  for  research 
or  operational  development  should  be  an  hour-for-houi  lesson  replacement.  This  avoids  implementation 
problems  of  scheduling  for  either  research  or  operatiomil  situations.  If  the  project  has  a research 
orientation,  the  hour-for-hour  lesson  replacement  allows  concise,  direct  comparison  of  the  different 
treatment  groups  on  identical  criterion  measures. 

Consideration  must  also  be  given  to  selection  of  courses  with  high  student  flov/.  In  today’s 
cost-conscious  Air  Force,  maximizing  the  number  of  students  administered  a teaching  method  with  a fixed 
cost  will  enhance  cost-effectiveness.  Kgh  student  flow  courses  should  not  be  extremely  time-pressured  or 
incapable  of  rhultiple  entry  points  because  of  difficulties  with  scheduling  classes. 

Once  these  criteria  are  used  to  identify  potential  lessons,  close  coordination  and  communication  is 
necessary  to  insure  that  the  staff  of  the  potential  course  views  CAl  as  a valuable  asset  rather  than  a 
potential  threat.  The  staff  should  realize  Aat  they  will  be  integrally  involved  in  lesson  development, 
selection  and  review  of  material,  and  test  item  approval.  This  close  working  relationship  facilitates  quality 
lesson  development  plus  establishes  a sound  basis  for  integrating  and  implementing  a viable  program. 

Before  lesson  development  begins,  evaluation  of  student  characteristics  and  criterion  measures  should 
be  accomplished.  Criterion-related  student  characteristics  assist  CAl  authors  in  the  development  of 
instructional  materials  and  strategies  appropriate  to  the  target  population.  Subsequent  validation  of  these 
materials  and  strategies  is  therefore  based  upon  a student  profile  of  characteristics  which  are  predictive  of 
performance  levels.  Instructional  revision  or  compensatory  alternatives  may  then  be  developed  and 
differentially  assigned  to  students,  resulting  in  improved  student  attainment  of  instructional  objectives  in 
less  time.  This  approach  thus  affords  a greater  degree  of  individualized  instruction.  Similarly,  criterion 
performance  measures  must  be  closely  examined  for  validity  to  ensure  consistency  between  the  material 
taught  and  the  performance  standards  required.  Research  questions  bearing  on  the  difference(s)  in  relative 
instructional  effectiveness  of  two  or  more  instructional  methods;  e.g.,  the  POMC  as  opposed  to  lecture,  are 
more  likely  to  be  answerable  given  criteria  Jiat  accurately  and  reliably  measure  performance  standards. 
Accordingly,  performance  measures  which  distinguish  successfully  trained  students  from  students 
unexposed  to  the  instructional  material  are  recommended  as  one  approach  for  documenting  adequate 
criteria. 

Adequate  time  for  conducting  systematic  formative  evaluation  is  not  only  instructionally  important, 
it  is  a pre-implementation  phase  investment  which  minimizes  total  time  expended  in  correcting 
instructional,  procedural,  and  attitudinal  problems  during  subsequent  operational  implementation.  The 
formative  evaluation  sequence  includes:  (a)  author-peer  review,  (b)  small  group  validation,  and  (c)  large 
group  validation  employing  representative  samples  of  students  who  have  not  been  previously  exposed  to  the 
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material,  llie  foregoing  sequence  with  instructional  revision  as  required,  increases  the  likelihood  of  effective 
instruction  and  decreases  the  probability  of  subsequent  m^or  problems  during  the  implementation  and 
comparative  evaluation  stages. 

Experience  of  the  PLATO  IV  sta^  at  Sheppard  with  the  technique  of  assigning  some  members  of  the 
class  to  an  instructiona’  treatment  on  one  set  of  instructional  objectives  which  differ  from  what  student 
counterparts  in  the  conventional  class  experience,  results  in  student  confusion,  rumort  about  “missing” 
material,  perceived  extra  workloads  and  complications  in  grading  procedures. 

The  foregoing  suggestions  and  recommendations  were  derived  from  the  collective  experiences  of 
military  health  care  professionals  who  composed  the  PLATO  IV  staff  (Appendix  H)  at  Sheppard  AFB, 
Texas.  Detailed  rationale,  procedures,  and  analyse:  are  found  in  the  main  body  of  the  report  which 
hopefully  will  be  of  beneht  to  health  cate  penonnel  contemplating  problem-oriented  medical  training  by 
means  of  computer  interactive  terminala. 
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APPENDIX  A:  RESEARCH  AGREEMENT 
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1.  TITLE:  Evaluation  of  PLATO  IV  System  in  Air  Force  Medical  Training 

2.  OBJECTIVES:  The  two  major  objectives  of  this  study  are:  (a)  the  research  and  development  of  a new, 
problem-oriented,  medical  curriculum  in  the  Physician  Assistant  Course  using  an  advanced,  computer-based 
educational  system  and  (b)  a comparative  evaluation  of  the  instructicmal  effectiveness  of  this  curriculum 
vis-a-vis  the  present  traditional  curriculum. 

3.  PROPOSED  MILESTONE  SCHEDULE: 

a.  Start  Date:  Nov  1973 

b.  Completion  Date:  lun  1976 

c.  Hnal  Report  Date:  Nov  1976 

4.  SUPPORT  TO  BE  FURNISHED  BY  ATC: 

a.  Provide  adequate  classroom  and  laboratory  space  to  acconunodate  20  PLATO  IV  terminals  and 
associated  equipment  at  the  School  of  Health  Care  Sciences,  Sheppard  AFB,  Texas. 

b.  Provide  students  for  the  conduct  of  the  PLATO  evaluation. 

c.  Provide  subject  matter  specialists  and  technical  support  personnel  to  develq>  200  hours  of  Riysician 
Asastant  Course  materials  for  implementation  on  the  ^ATO  system,  to  train  instructors  in  the  use  of 
PLATO  at  Sheppard,  to  review  existing  and  developing.  University  of  Illinois,  PLATO  programs  in  the 
health  cate  sciences  area,  and  to  assist  in  the  data  collection,  course  evaluation  and  d rcumentation.  To  the 
maximum  extent  possible  all  penonnel  assigned  to  die  PLATO  IV  program  at  Sheppud  will  have  sufficient 
tenure  to  see  the  program  through  its  planned  three-year  time  span.  PLATO  IV  duties  will  constitute  the 
primary  responsib^ty  of  the  personnel  assigned  to  the  experimental  program. 

d.  Document  the  development  procedures  used  and  lessons  learned  for  future  reference. 

e.  Develop  a test  plan  for  the  evaluation  of  PLATO  IV  with  AFHRL/TT. 

f.  ftovide  a monthly  report  of  progress  to  AFHRL/TT,  HQ  ATC/XPT,  SG,  and  Advanced  Research 
Projects  Agency  (ARP A). 

5.  SUPPORT  TO  BE  FURNISHED  BY  AIR  FORCE  SYSTEMS  COMMAND  HUMAN  RESOURCES 
LABORATORY  (AFHRL/TT): 

a.  Provide  direction  and  monitoring  of  the  PLATO  IV  evaluation. 

b.  Develop  a test  plan  for  evaluatio.i  of  PLATO  IV  in  conjunction  with  SHCS  representatives. 

c Derive,  implement,  and  monitor  measures  for  student  and  instructor  attitudes  as  wdl  as  student 
performance. 

d.  Consult  on  instructional  systems  development,  instructional  strategies,  instruction^  materials,  and 
media  devices. 

e.  Plan  for  the  incorporation  of  computer  based  educational  materials  developed  at  SHCS  into  the  AIS 
following  completion  of  the  evaluation  effort.  If  feasible,  the  materials  developed  for  the  Physician 
Assbtanfs  Course  will  be  integrated  into  and  tested  with  the  AIS  management  component. 
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f.  Consult  on  the  use  of  PLATO  and  the  TUTOR  piogranuning  language. 

g.  Serve  as  the  primary  point  of  contact  to  ARPA  on  all  matters  pertaining  to  the  University  of  Dlinois 
contract. 

h.  Provide  quarterly  reports  on  progress,  an  annual  summary  report  of  progress  and  findings,  and  a 
final  report  at  the  conclusion  of  the  experimental  test  to  ARPA  and  ATC. 

6.  RESOURCES  AND  SUPPORT  TO  BE  FURNISHED  BY  THE  ADVANCED  RESEARCH  PROJECTS 
AGENCY  (ARPA)  THROUGH  CONTRACT  WITH  THE  UNIVERSITY  OF  ILIJNOIS: 

a.  Procurement,  installatior,  and  maintenance  at  the  School  of  Health  Care  Sciences,  Sheppard  AFB, 
Texas,  of  20  FlATO  IV  termin  Js. 

b.  Computer  support,  software,  and  communications  services  essential  to  providing  a comf^ete  PLATO 
IV  capability  to  remote  terminals  at  Sheppard  AFB,  Texas. 

c.  Training  of  ATC  persormel  in  the  TUTOR  language,  curriculum  development,  operating  dynamics, 
and  instructional  capabilities  of  the  PLATO  IV  system. 

d.  Advisory  assistance  throughout  the  three  year  contract  period  in  curriculum  development  and  use 
of  the  PLATO  IV  system. 

e.  Cooperative  development  of  curriculum  materials  for  the  project  including  the  review  of  other 
materials  on  the  PLATO  system,  conversion  of  selected  materials  to  fit  the  Sieppard  requirements,  and  the 
development  of  selected  course  materials  and  support  modules  as  agreed  to  with  the  Sheppard  project 
manager. 

7.  RATIONALE  RELATING  TO  PLATO  IV  EVALUATION  EFFORT: 

a.  A basic  core  of  medical  knowledge  for  USAF  Physician  Assistants  exists  as  outlined  in  the  Han  of 
Instruction,  Course  3ALR91730,  dated  3 August  1972. 

b.  A basic  pattern  of  clinical  behavior  characterizes  problem-solving  which  is  straightforward,  logical, 
and  scientific. 

c.  A charting  system  exists  for  recording  and  documenting  the  use  of  medical  knowledge  in  the  solving 
of  clinical  problems. 

d.  Advances  in  educational  technology  and  instructional  systems  analysis  can  be  greatly  uded  by 
computer  support  which  integrates  and  correlates  7(a)  and  (b). 

e.  Such  a system  will  be  individualized  and  learner  oriented.  Evaluative  systems  will  constantly  assess 
instructional  effectiveness,  student  achievement  level,  and  resource  impact.  Quality  instruction  and 
performance  will  not  be  compromised  in  deference  to  resource  conservation. 

8.  APPROACH: 

a.  Approximately  200  hours  of  instructional  material  will  be  developed  by  SHCS  personnel.  Course 
development  will  be  in  accordance  with  provisions  of  AFM  50-2. 

b.  During  the  evaluation  effort,  students  will  be  exposed  to  both  curricular  formats  in  which  PLATO 
IV  may  or  may  not  be  employed.  The  suitability  and  contribution  of  computer  based  instruction  will  be 
evaluated  as  it  stands  alone  and  vis-a-vis  the  curricula. 

c.  The  experimental  PLATO  curriculum  will  be  developed  and  administered  by  a specifically 
designated  unit  within  SHCS. 
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9.  DISPOSITION  OF  RESOURCES  AFTER  COMPLETION  OF  PROJECT;  Upon  the  completion  of  the 
4ttee-year  experimental  service  test,  all  PLATO  IV  equipment  will  be  retained  by  the  School  of  Health  Care 
Sciences  and  become  the  property  of  die  Air  Force.  These  resources  will  provide  the  Air  Force  with  a 
curriculum  development  laboratory  for  computer*based  instruction  in  Air  Force  medical  training. 


id.  This  research  agreement  has  been  drafted  in  accordance  with  ATCR  80-1.  The  provisions  and 
re^nsibilities  listed  have  been  entered  into  by  all  parties  in  good  faith  and  in  recognition  of  the  exigencies 
of  the  Air  Force. 


AGREED  TO  BY; 


DATE 


ter  _i£_5££ip 


ARPA 


DATE 


DATE 
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APPENDIX  B:  POMC  IMPLEMENTATION  PLAN 


I.  References:  Course  Chart  3ALR91730 

POI  3ALR91730 

Research  Agreement  "Evaluation  of  PLATO  IV 
System  in  Air  Force  Medical  Training" 

II.  Plan  Objective 

The  purpose  of  this  plan  is  to  provide  for  the  orderly 
integration  of  the  Problem  Oriented  Medical  Ctirriculum  (POMC) 
developed  by  the  PLATO  IV  Development  Branch  ir.to  the  present 
Physician  Assistant  curriculum.  This  integration  will  be  done 
in  such  a way  that: 

1.  Disruption  of  the  present  curriculiim  is  minimized. 

(a)  Steps  will  be  taken  to  insure  that  prerequisite 
material  is  taught  prior  to  the  time  it  is  required  as  a basis 
for  other  material. 

(b)  The  logical  flow  of  content  in  discipline  oriented 
blocks  will  not  be  jeopardized. 

2.  The  integrity  of  the  POMC  is  maintained,  i.e.  presenta- 
tion of  material  is  not  excessively  fragmented. 

3.  No  degradation  in  training  will  result.  ^ 

4.  Evaluation  of  the  POMC  is  facilitated. 

III.  Scope 

This  plan  pertains  only  to  the  Physician  Assistant  Course 
(Phase  I),  3ALR91730.  The  time  frame  for  implementation  of  this 
plan  is  Jtin  1975  - Oct  1976  to  correspond  with  ♦•he  evaluation 
phase  of  the  PLATO  IV  project. 


IV.  Background 

k.  On  4 Apr  73,  the  School  of  Health  Care  Sciences  sub- 
mitted a proposal  for  personnel  research  to  Hq  ATC.  The  re- 
sultant resecirch  agreement  calls  for  (1)  the  research  and 
development  of  approximately  200  hours  of  a new  problem  oriented 
medical  curriculum  at  the  physician  assistant  level,  using  an 
advanced  computer  based  education  system,  and  (2)  the  < valuation 
of  the  instructional  effectiveness  of  this  curriculim  vis-a-vis 
the  traditional  curriculum.  The  project  covers  a three  year 
period  beginning  in  November  1973.  Evaluation  is  to  be  completed 
by  1 July  1976,  with  the  final  report  to  be  rendered  in  November 
1976. 

B.  A problem  oriented  medical  curriculum  Is  defined  as  the 
grouping  of  course  material  under  clinical  topics  where  the 
clinical  relevance  of  cotirse  content  can  be  stressed.  Emphasis 
Is  placed  on  developing  problem  solving  skills  in  diagnosis  and 
treatment  of  disease.  To  facilitate  development,  the  POMC  has 
been  divided  into  two  parts.  Part  I presents  the  relevant  basic 
science  and  clinical  material  in  an  integrated  manner  supported 
by  illustrative  case  material  and  using  clinical  subjects  as 
point  of  departxure  to  teach  basic  sciences.  Part  II  will  con- 
sist of  "Simulated  Patient  Encountersi"  which  will  be  approached 
as  \mknowns  by  the  students.  Diagnostic  and  treatment  skills 
as  well  as  clinical  problem  solving  skills  will  be  amplified 
in  Part  II. 
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C.  The  task  of  selecting  the  content  of  the  POMC  from  the 
massive  amount  of  material  included  in  the  present  physician 
assistant  course  was‘  tindertaken  as  follows.  Selection  was  based 
on  the  criteria  that  the  clinical  area  would  (1)  accommodate  an 
integrated  approach  to  include  both  the  basic  and  clinical 
sciences,  (2)  contain  fairly  circumscribed  and  clinically  rele- 
vant subject  matter,  (3)  provide  sufficient  subject  matter  to 
meet  the  project  objectives,  and  f4)  make  the  best  use  of  re- 
sources within  the  PLATO  IV  Development  Branch.  Jointly  ex- 
haustive lists  of  possible  etiological  and  body  systems  were 
tested  against  the  selection  criteria.  The  respiratory  system 
(respiratory  disease)  was  selected  as  the  best  subject  area  for 
the  purposes  of  this  project. 

D.  A list  of  basic  sciences  and  specific  clinical  topics 
has  been  developed  and  arranged  in  a curricular  structure  which 
defines  the  basic  sequence  and  interrelationships  of  the  subject 
matter  areas.  Initial  planning/programming  has  begim  in  eight 
of  the  proposed  20  lesson  development  areas. 

V.  Approach  to  Implementation 

A.  Definitions  and  Constraints 

1.  'Terminals.  There  are  16  PLATO  IV  student  terminals. 
Prime  time  (when  the  system  is  most  stable)  is  from  0740  to  2200 
hours,  Monday,  Wednesday  and  Friday,  0600  to  2200  Tuesday  and 
Thursday,  and  0800  to  1200  hours  Saturday.  The  total  available 
terminal  hours  is  a constraint,  since  there  are  64  PA  students 
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in  any  given  semester,  plus  30  PCNP  students  who  attend  PA 
clinical  medicine  lectures. 

2.  POI  Hours.  For  planning  purposes,  hours  are  con- 
sidered to  be  POI  hours  planned  for  a particular  subject.  POI 
hours- may  not  necessarily  equate  to  hours  actually  spent  at  the 
terminal  by  a given  student.  PLATO  material  at  the  lesson  or 
lesson  sequence  level  is  self-paced  and  variations  in  individual 
students  can  be  expected. 

3.  Classroom  (C/L) /Complementary  Technical  Training  (CTT) 
Hours.  Since  computer  assisted  instruction  is  designed  to  teach 
for  mastery  of  the  material,  a PLATO  IV  lesson  designed  to  meet 

a particular  POI  objective  will  replace  both  the  associated 
classroom  hours  and  CTT  hours.  In  computing  hours  in  the  present 
curriculxim  to  be  replaced  by  PLATO  material,  both  C/L  and  CTT 
hours  are  considered. 

B.  Hours  Required  for  POMC  Material. 

1.  The  present  PA  curriculum  has  been  examined  to  identify 
those  hours  in  which  the  subject  matter  is  essentially  the  same 

as  POMC  content  and  can  therefore  be  replaced  by  the  comparable 
POMC  material.  A detailed  breakdown  of  hours  by  POI  objective  by 
semester  is  included  in  Attachment  1.-  A summary  is  provided  in 
paragraph  3 below. 

2.  As  anticipated,  some  material  included  in  the  POMC 
does  not  have  direct  comparability  with  hours  in  the  present 
curriculum.  This  material  is  limited  to  "simulated  patient 
encounters"  which  is  expected  to  require  24  student  contact  hours. 
These  hours  are  considered  to  be  an  essential  part  of  the  POMC 
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for  review  and  reinforcement. 

3.  Summary  of  Hours 


9 


Replacement  Hours : 


C/L 

CTT 

Sub -Total 

Total 

1st  trimester 

16.5 

7.5 

24 

2nd  trimester 

20.0 

5.0 

25 

3rd  trimester 

52.0 

20.0 

72 

Sub-total 

88.5 

32.5 

121 

121 

Additional  Hours; 

Simulations 

24 

Sub-total  24 

Grand  total : 145 

4.  In  the  original  research  proposal  an  arbitrary 
estimation  of  200  course  hours  was  designated  for  PLATO  IV 
programs.  The  reduction  in  these  hours  to  145  represents  the 
integration  of  an  effective  circumscribed  curriculxom  into  tbe 
present  course  with  the  least  disruption  of  continuity.  The 
expansion  of  the  PLATO  hours  to  reach  a closer  approximation 
of  the  original  estimation  of  hours  would  only  serve  to  dilute 
the  effectiveness  of  the  PLATO  project,  complicate  the  evalu- 
ation procedures  and. increase  the  problems  of  integration  into 
the  present  course. 

C.  Plan  for  Implementation 

1.  A sample  of  16  Physician  Assistant  students  will 
be  selected  from  the  freshman  class  starting  in  June  1975  to 
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serve  as  a pilot  group.  Sampling  techniques  developed  by  AFHRL 
will  be  used  to  select  the  16  students.  A second  sample  of  16 
students  may  be  selected  from  the  class  starting  in  Oct  1975  if 
that  is  considered  desirable  on  the  basis  of  evaluation  plans 
and  experience  with  the  initial  group. 

2.  Students  in  the  sample  will  complete  PLATO  materials 
in  lieu  of  the  corresponding  work  in  the  current  curricultam. 

They  will  attend  all  of  the  scheduled  lectures,  except  those  that 
have  been  replaced  by  PLATO  lessons.  In  addition  to  criteria 
listed  in  IV-C  above,  considerable  concessions  have  been  made  in 
the  selection  of  material  to  be  included  in  the  POMC  so  that 
discrete  lecture  hours  of  the  current  curriculum  are  identifiabli 
At  the  time  such  lectures  are  given,  students  would  leave  the 
classroom  and  report  to  the  PLATO  lab  for  computer  based  instruc 
tion.  Overall,  the  same  criterion  objectives  will  be  met  by  all 
students,  but  the  temporal  sequencing  of  material  in  the  problem 
oriented  medical  curriculum  will  not  necessarily  parallel  that  o 
the  current  curriculum. 

3.  The  additional  hours  required  for  the  simulated 
patient  encounters  Included  in  the  POMC  will  be  supplemental 
hours  to  be  done  at  the  student’s  convenience.  This  will  provide 
review  and  reinforcement  of  material  covered  in  Phase  I. 

4.  Since  students  in  the  sample  group  will  not  neces- 
sarily be  working  on  the  same  subject  matter  at  the  same  time  as 
the  rest  of  the  class,  some  modification  of  the  normal  end  of 
block  evaluation  procedures  will  be  required.  POMC  students  wov; 
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take  the  normal  end  of  block  tests,  but  questions  covering 
coursework  taught  in  the  POMC  would  not  be  covinted  in  their 
test  score.  The  PLATO  IV  staff  will  assume  responsibility  for 
determining  student  proficiency  in  the  POMC  subject  areas  on  a 
semester  basis  to  determine  whether  or  not  the  student  is  com- 
petent to  proceed,  needs  remediation,  etc.  The  PLATO  staff  will 
also  assum.'!  responsibility  for  needed  remediation  in  those  areas. 
VI.  Tasks  Remaining  to  be  Done. 

A.  A detailed  schedule  must  be  developed  for  the  present 
curriculim  that  identifies  specific  subjects  for  specific 
lecture  hours,  and  also  identifies  hours  during  which  POMC  material 
is  to  be  taken  by  the  sample  group. 

B.  Waivers  from  course  doctmientation  requirements  and 
testing  standards  must  be  requested  from  Hq  ATC. 

C.  Any  questions  involving  accreditation  from  the  University 
of  Nebraska  for  the  sample  group  must  be  resolved. 

1 Atch 

Breakdown  of  Replacement  Hours 
by  POI  Objective 
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21k  Describe  the  physical  signs 

and  symptoms  of  aerobic  and 
anaerobic  bacterial  pneumonias, 
and  mycotic  diseases  of  the 
lungs . 
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problems  in  childhood,  their  piratory  dis.  of 

evaluation  and  management  childhood) 


APPENDIX  C:  FLOW  CHART/POI  OBJECTIVES 


FLOW  CHART  BLOCK 

1.  Introduction 
POI  Obj: 


2.  Basic  Sciences 

POI  Obj:  VI-6a,  X-6a, 
XII-6b,  XIII-3a 

Anat  Term 


FLOW  CHART  BLOCK 

8.  Allergy/Antihistamines 
POI  Obj : XXI-9,  XX-4a 

Allergy 

Allergens 

Antihistamines 


Intro 

Anatomy 

Flnctional  Mechanics 
Dif&ision 
Transport 
Control 

3.  Upper  Aixvay  Disease 
POI  Obj:  XIV-6b 


4.  History  in  Respiratory 
Disease;  POI  Obj:  Xn-21a 


6.  Problem-Oriented 
Medical  Record 
POI  Obj:  XXI-21a 


7.  Procedures/Lab  Tests 

POI  Obj;:  IX-4,  IX-6a,  lX|-8 


9.  X-Ray 

POI  Obj : XEC-2a-n 


10.  Pathophysiology  of 
Inflammation 
POI  Obj:  XXI-21b, 
VIII-llb,-llc,-lle 

Pathophysiology 

Heterophile 

ASO 

Febrile  Agglut. 


11.  Infiltrative  Infectious 
Lung  Disease;  POI  Obj: 

XXI -13b,  Xn-21j,  -21k,  -21o,  -21g 

Introduction 

Lung  Abscess 

Pneunonia 

Tbc 

Interstitial  pneumonia 
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FLOW  CHART  BLOCK 


FLOW  CHART  BLOCK 


12,  Infiltrative  Noninfectious  15.  Pulmonary  Function 
Lung  Disease  - POI  Obj:  POI  Obj:  XXI -21c 

XXI-21i,  -21m,  -21a,  -21p  

16.  fiiiboli 

POI  Obj : XXI-21£ 


17.  Neoplasm 

POI  Obj : XXI-211 

Neoplastic  Disease 

Cytology 


13.  Antibiotics  18.  Flight  Considerations 

POI  Obj;  XX-22a  POI  Obj:  XVIII-la 


14.  COPD 

POI  Obj:  XXI-21d, 
-21h 

Introduction 

OOPD 

Asthma 

Bnphysema 

Chronic  Bronchitis 


19.  ARF/Noxious  Drugs 
POI  Obj ; XXI-21C 

ARF 

Noxious  Drugs 


20.  Simulations 

POI  Obj : m-13d 

Mono 

Other 


50 


r 


APPENDIX  D:  SIMULATED  PATIENT  ENCOUNTERS  USING 
THE  PLATO  IV  COMPUTER-BASED  EDUCATIONAL  SYSTEM 


Divid  R.  Stutz,  Mtjor.  USAF.  MC 
and 

EImd  Sweeney,  B.A. 
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Major  Stutz  ii  die  Medical  Director  of  the  USAF  School  of  Health  Care  Sciences 
PLATO  IV  Development  Project  at  Sheppard  AFB,  Texas.  Ms.  Sweeney  is  on  the 
staff  of  CERL,  Computer-based  Education  Research  Laboratory,  at  Champaign, 
Illinois,  where  ^ is  director  of  lesion  review  for  the  Sheppard  Proj^. 
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ABSTRACT 


The  PLATO  IV  Development  Project  at  Sheppard  AFB,  Texas,  is  designing  a minicurriculum  on  the 
respiratory  system  and  its  diseases  for  physician  assistant  students.  A mqor  portion  of  this  curriculum  is 
devoted  to  simulated  patient  encounters  presented  via  the  PLATO  IV  computer-based  instructional  system. 
These  simulated  patient  encounters  are  designed  to  provide  a model  for  systematic  data  collecuoii  and  its 
analysis,  and  at  the  same  time  reinforce  the  instruction  on  respiratory  system  diseases. 


The  PLATO  IV  Development  Project  at  the  USAF  School  of  Health  Care  Sciences  is  a minicuniculum 
on  “The  Respiratory  System  and  Its  CKseases,”  implemented  for  physician  assistant  students  by  using  the 
PLATO  IV  computer-based  instructional  system.'  This  is  a system  of  remote  terminals  operating  throu^  a 
central  computer  at  the  University  of  Illinois.^  A major  component  of  the  instructional  lessons  is  a series  of 
simulated  patient  encounters  (“Simulations").  This  paper  will  describe  a typical  simulation,  showing  how  it 
is  used  as  a model  for  systematic  data  collection  and  iu  analysis,  and  how  it  serves  as  a vehicle  for 
presenting  clinical  medical  material. 


STUDENT  ORIENTATION 

Prerequisite  to  the  first  simulation  is  a lesson  on  “History-Taking  in  Pulmonary  Diseases.”  It  uses  the 
same  format  as  the  simulations  that  follow,  with  the  student  typing  in  questions  and  seeing  immediate 
“patient”  responses  on  the  remote  terminal’s  piasttui  panel.  This  lesson  serves  as  a model  for  communicating 
with  the  simulated  patients,  and  it  develops  the  following  points  as  the  student  is  led  through  the 
history -taking  exercise: 

Wr’hen  encountering  a patient  with  suspected  pulmonary  disease,  one  must  inquire  about  the  following 
cardinal  symptoms: 

Chest  pain  Cyanosis 

Dyspnea  Wheezing 

Cou(^  and  Hemoptysis 

For  any  positive  response,  the  student  is  told  to  amplify  the  description  of  that  symptom  by  asking 
about: 

Provocative  and  palliative  factors. 

Quality  (precise  nature)  of  the  symptom. 

Region  of  the  body  involved. 

Severity  of  the  symptom,  and 

Temporal  factors. 

This  is  reinforced  by  the  mnemonic  "PQRST.”* 

In  addition,  the  student  learns  to  ask  about:  smoking  habits,  allergies,  occupation,  and  travel,  since 
these  factors  can  be  related  to  respiratory  system  disease. 

Before  “seeing”  the  simulated  patient,  the  student  will  also  have  completed  lessons  covering  the 
physical  exam  of  the  respiratory  system,  laboratory  studies  in  respiratory  disease,  and  the  problem-oriented 
medical  record. 


TYPICAL  PATIENT  ENCOUNTER 


The  student  initiates  the  interaction  in  the  history-taking  mode,  but  at  any  time  he/she  may  branch 
to  the  physical  exam  or  laboratory  exam  mode  to  collect  pertinent  information.  In  each  of  these  areas, 
assistance  is  available  which  summ‘*rizes  the  prior  lesson  on  history-taking,  or  which  details  the  list  of 
available  laboratory  and  other  diagnostic  procedures.  The  student  is  free  to  branch  back  and  forth  among 
the  various  data  collection  modes,  and  at  any  time  he/she  may  elect  to  make  a diagnosis. 

When  asking  questions  to  the  simulated  patients,  the  student  may  use  spontaneous  and  natural 
language.  Warning  Is  given  that  only  one  question  should  be  asked  at  a time  and  ^at  questions  should  be 
specific  and  not  leading. 

The  patient  is  presented  initially  as  follows: 

Computer:  A 24  year  old  male  sergeant  presents  to  the  emergency  room. 

Initial  observation  by  you  reveals  a well-nourished  man  who  is  sitting  very  quietly  in  his  chair. 

A typical  dialogue  is  described  below: 
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Student:  Wh  s the  reason  you  have  come  here? 

Patient:  ljus:  n’t  get  my  breath  and  my  chest  hurt$. 

Student:  Wheu  i»  the  chest  pain? 

Patient:  The  pain  is  here  in  my  left  side,  and  1 also  feel  it  in  my  left  shoulder. 

Student:  What's  the  chest  pain  like? 

Patient:  The  pain  is  like  a knife  in  my  side.  It  really  hurts  if  I take  a deep  breath, 

f :iuli.nt  Does  it  conic  on  when  you  bend  over? 

l*atient:  Sorry,  but  1 don’t  understand  your  question.  (Please ...  avoid  pronouns.  Try  to  be  more 

specific  or  phrase  the  question  differently.  Also ...  ask  about  one  thing  at  a time) 

-HELP-  is  available 

Student:  What  brings  on  the  chest  pain? 

Patient:  The  pain  came  on  by  itself,  but  it’s  worse  when  I take  a deep  breath. 

Student:  What  relieves  your  problem? 

Patient:  I just  try  to  sit  still. 

Student:  Have  you  taken  any  medication  for  this? 

Patient:  I haven’t  taken  anything  for  it. 

At  any  time  the  student  may  branch  to  the  physical  exam  mode  to  “perform”  components  of  that 

exam. 

Student:  Vital  signs 

Response:  T.99*  P 104  (reg)  R 20 (reg)  BP  120/85 
Student:  Chest  exam 

Response:  INSPECT:  Rapid  rate  (20)  without  the  use  of  accessory  muscles.  Left  side  does  not  appear  to 
expand  as  well  as  the  right.  PALPATION:  Fremitus  decreased  left  lateral  and  posterior  lung 
fields.  PERCUSS:  Increased  resonance  over  left  lateral  and  posterior  lung  fields. 
Diaphragmatic  excursion  normal  bilaterally.  AUSCULTATION:  Decreased  breath  sounds  over 
left  lateral  and  posterior  lung  fields. 

The  student  is  also  free  to  request  laboratory  data  at  any  time.  For  example: 

Student:  Lung  scan 

Response:  Lung  scan  is  utmecessary  if  you  bothered  to  order  a chest  x-ray  first. 

Student:  Chest  x-ray 

Response:  Chest  x-ray,  X-ray  #B-3  to  be  used.  Press  SHIFT-LAB-  for  assistance  interpreting  the  chest 
x-ray.* 

The  student  may  also  jlect  to  make  a diagnosis  at  any  time,  but  before  he/she  is  allowed  to  enter  a 
diagnosis,  certain  specified  data  must  have  been  collected.  If  the  specified  information  has  not  been 
obtained,  the  student  receives  this  message  when  he/she  elects  to  enter  a diagnosis: 

Computer:  You  do  not  have  all  of  the  information  necessary  for  making  a diagnosis.  Before  you  are 
allowed  to  attempt  a diagnosis,  you  should  investigate  the  following  areas: 

*“SH1I  T-l  AB  - ” refers  to  a key  on  the  PLATO  terminal's  keyboard. 
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When  did  the  problem  start? 

Is  there  any  cou^? 

The  student  cannot  proceed  until  the  specified  questions  am  asked.  Similarly,  after  the  required 

history  information  is  collected,  if  there  is  necessary  physical  or  laboratory  exam  data  that  is  not  yet 

collected,  the  student  will  receive  the  appropriate  message. 

ASSESSMENT  OF  THE  DATA 

Once  in  the  diagnosis  section,  the  student  is  allowed  to  enter  as  many  diagnoses  as  he/she  wishes. 

Computer:  Enter  as  many  diagnoses  as  you  wish,  but  press  -NEXT-  after  each  entry.  Then  press 
—DATA—  when  you  have  pven  all  the  diagnoses  you  feel  are  relevant.* 

Student:  Pulmonary  Embolism 

Computer:  Although  it  is  diffica’'  to  rule  out  this  diagnosis,  the  correct  diagnosis  can  be  made  from  the 
chest  x-ray.  There  is  m reason  to  suspect  pulmonary  embolism  as  well. 

The  student  is  free  to  return  to  the  data  collect  j.i  mode  to  review  the  data  that  has  been  collected. 

When  in  the  diagnosis  mode,  assistance  is  available  in  the  form  of  a “Socratic”  type  question  and 

answer  exchange.  The  student  may  access  this  by  pressing  the  appropriate  key  on  the  keyboard  (labeled 

“HELP’). 

Computer:  A key  to  diagnosis  in  this  case  is  the  proper  interpretation  of  the  chest  x-ray.  Look  at  chest 
x-ray  #B-3.  You  should  note  that  this  is  an  expiratory  phase  film  which  is  the  best  type  of  film 
to  diagnose  this  condition.  First,  look  at  the  RIGHT  lung  field  and  notice  that  the  vascular 
markings  extend  ail  the  way  to  the  periphery.  Is  that  normal? 

Student:  Yes 

Computer:  Of  course  it  is.  Although  the  mairkings  become  finer  as  tlie  pulmonary  vessels  branch  away  from 
the  hilum,  some  markings  will  be  present. 

Computer:  Now  look  at  the  superior  and  lateral  portion  of  the  left  lung.  Although  it  is  difficult  to  see  m 
this  small  chest  x-ray,  note  that  beyond  the  dashed  line,  no  lung  markings  are  present.  Whai  is 
present  between  the  dashed  lines  and  the  parietal  pleura  which  lines  the  thoracic  cage? 

Student:  I don't  know. 

Computer:  The  substance  present  is  one  of  the  classical  states  of  matter. 

Student:  liquid 

Computer:  That  would  be  denser.  A lucency  is  present.  Try  again. 

Student:  Gas 

Computer:  Correct.  Air  is  present  in  the  pleural  space. 

Computer:  What  is  the  term  for  air  in  the  pleural  space? 

Student:  You  got  me. 

Computer:  The  first  part  of  the  word  is  from  the  Greek,  ‘'PNEUM(J’  meaning  air. 

Student:  I still  don’t  know. 

Computer:  The  second  part  of  the  word  refers  to  the  chest  cavity. 

Student:  Ihieumothorax. 

• -NEXT-  and  -DATA  - refer  to  particular  keys  on  the  PLATO  terminal's  keyboard. 
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Computer:  That’s  ri^t.  The  patient  has  a partial  pneumothorax.  It  is  partial  because  most  of  the  left  lung 
is  still  expanded. 

After  an  of  the  diagnoses  are  made,  the  student  proceeds  to  a section  to  determine  the  etiology  of  the 
problem.  Like  the  diagnostic  cunsultation,  this  section  is  presented  via  an  interactive  dialogue  between  the 
computer  and  the  student. 

After  this  section  is  completed,  the  case  is  summarized  by  using  the  problem-oriented  record  “SOAP’ 
technique  as  described  below:^ 

#1  Pneumothorax 

SUBJECTIVE:  (History  data  is  summarized.) 

OBJECTIVE:  (Physical  and  lab  exam  data  is  sununatized.) 

ASSESSMENT:  ^eumothorax,  etiology-spontaneous. 

Approximately  20%  of  left  lung  involved. 

PLAN:  Press  —NEXT—  to  formulate  a plan. 


DEVELOPMENT  OF  THE  PLAN 

This  section  is  conducted  in  the  same  way  as  the  earlier  sections.  The  student  develops  his/her 
diagnostic  plans  (further  studies)  and  treatment  plans  (induding  follow-ups),  and  after  this  is  done,  a 
completed  “SOAP”  summary  is  shown,  now  including  the  plan.  Depending  on  the  choices  made,  the 
student  may  have  achieved  a therapeutic  success  or  failure,  and  appropriate  feedback  is  given. 


DIDACTIC  MATERIAL 

After  completion  of  the  case,  a descriptive  summary  of  the  disease  entity  is  presented.  This  covers: 

Eipidemiology:  who  is  at  risk. 

Signs  and  Symptoms 
Pathophysiology 

Special  studies  required  to  make  the  diagnosis  and  to  monitor  progress. 

Treatment,  Prognosis,  and  Prophylaxis* 

! 

This  is  a format  that  the  student  can  adopt  to  store  and  retrieve  information  in  and  from  his  own  mind. 


CONCLUSION 

Althou^  the  simulated  patient  encounten  caimot  duplicate  the  subtleties  ofinteracting  with  a living 
patient,  the  simulations  and  the  prerequisite  lessons  on  history-taking,  physical  exam,  and  the 
problem-oriented  medical  record,  can  do  the  following: 

1.  They  provide  an  explicit  model  for  systematic  history  data  collection,  helping  to  insure  a 
complete  description  of  any  given  symptom. 

2.  They  provid,e  an  implicit  model  for  analyses  of  the  collected  data  by  the  use  of  interactive 
dialogues  between  the  mentor  (computer)  and  the  student. 

3.  They  reinforce  the  use  of  the  problem-oriented  medical  record  by  using  the  “SOAP”  format.  This 
is  excellent  for  organizing  clinical  data,  impressions,  and  plans. 

4.  Finally,  they  present  material  describing  disease  entities,  using  a systematic  format  diat  the 
student  can  also  use  to  store  and  retrieve  descriptive  medical  material.  Thus,  the  lessons  provide  a model  for 
systematic  solving  of  medical  problems  as  weU  as  techniques  for  personal  storage  and  retrieval  of  medical 
material. 
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APPENDIX  E:  PROBLEM-ORIENTED  MEDICAL  CURRICULUM 
EVALUATION  PLAN 

(prepared  by  AF.RL/TT  as  a result  of  evaluation  conference  held 
6-7  Mar  75  at  Sheppard  AFB) 

BACKGROUND 

Under  the  auspices  of  the  Advanced  Research  Projects  Agency 
(ARPA)  and  Air  Training  Command  (ATC) , a Problem-Oriented  Medical 
Curriculum  supported  by  PLATO  IV  technology  is  being  introduced 
in  the  School  of  Health  Care  Sciences . PLATO  can  provide  individ- 
ualized instructional  material  presented  through  a computer  con- 
trolled plasma  panel.  The  essential  notion  behind  the  Problem 
Oriented  Medical  Curriculum  (POMC)  is  that  academic  subject  matter 
should  not  be  introduced  in  isolated  units,  but  rather  in  the  con- 
text of  clinical  diagnosis  of  actual  patient  problems.  In  this 
fashion,  it  is  anticipated  that  problem  solving  skills  will  be 
developed  to  a greater  degree,  that  retention  of  information  will 
be  enhanced,  and  that  student  motivation  will  be  increased.  Details 
covering  introduction  of  this  innovation  into  the  Physician's 
Assistant  Course  can  be  foxond  in  the  POMC  Implementation  Plan,  10 
Feb  75. 

PROBLEM 

The  goal  of  this  evaluation  plan  is  to  develop  a set  of  pro- 
cedures for  assessing  the  effectiveness  of  the  proposed  POMC. 
RATIONALE 

The  initial  step  in  the  process  is  to  define  a rationale  which 
underlies  and  supports  the  entire  evaluation.  Proper  evaluation  is 
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obviously  a complex  matter,  for  the  concept  of  "effectiveness"  Is 
clearly  multidimensional.  However,  two  sets  of  questions  seem  to 
be  generated  almost  immediately.  The  first  group  is  primary  and 
basically  operational  in  nature;  these  questions  are  designed  to 
create  a rational  policy  for  or  against  full  scale  Implementation 
of  the  innovation.  Simply  stated,  this  set  of  questions  asks: 

. can  it  be  done? 

. is  it  ins true tionally  effective? 

. what  does  it  cost  (to  institute,  to  operate,  to  maintain)? 

. is  it  reliable? 

A secondary,  or  research-oriented  group  of  questions  is  also 
of  interest.  These  are  directed  at  explaining  the  success  or  failure 
of  the  innovation  in  the  particular  environment  studied.  This  set 
consists  of  asking 

. how  was  it  done? 

. why  was  it  effective? 

. how  could  it  be  Improved  (in  terms  of  costs  lowered  or  ef- 
fectiveness increased)? 

These  latter  questions  are  not  unimportant,  but  higher  priority 
must  be  given  to  the  first  set  since  they  were  judged  to  be  more 
relevant  to  the  purposes  of  the  sponsoring  agencies  (ARPA  and  ATC) . 

To  some  extent  the  secondary  questions  will  be  addressed  in  the 
final  report,  especially  with  regard  to  describing  the  process  in- 
volved in  setting  up  the  POMC,  but  the  major  emphasis  of  available 
resources  is  being  placed  on  answering  the  first  group  of  questions. 
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Another  Issue  which  was  faced  in  planning  the  evaluation  con- 
cerns the  extent  to  which  comparative  measurement  will  be  made, 
as  distinguished  from* following  purely  descriptive  procedures. 

This  distinction  Is  analogous  to  the  difference  between  norms  ref- 
erenced and  criterion  referenced  measurement.  In  the  comparative 
approach,  procedure  A Is  compared  with  procedure  B on  various 
criteria  of  effectiveness,  while  In  the  descriptive  approach, 
effectiveness  Is  measured  with  respect  to  meeting  standards  or 
specifications  In  some  absolute  sense.  Each  approach  has  Its  own 
peculiar  advantages,  but,  where  feasible.  It  was  decided  to  use 
comparative  measures  as  much  as  possible  In  order  to  contrast  the 
POMC  approach  with  traditional  Instruction. 

Another  conceptiial  consideration  Important  for  this  rationale 
deals  with  focusing  on  which  aspects  of  the  situation  should  be 
studied.  Although  not  exhaustive  of  the  possibility <^s  Inherent 
In  this  situation,  three  major  aspects  were  Identified  for  Inten- 
sive study — students,  authors,  and  PLATO  IV  Itself. 

A final  concern  Involves  deciding  which  specific  characteristics 
and  variables  need  to  be  measured.  Since  student  and  author  per- 
formance are  crucial  to  judging  effectiveness,  these  naturally  are 
emphasized.  But  It  was  also  viewed  as  Important  to  measure  attl- 

I 

tudes  and  to  sample  opinions  from  the  participants.  In  addition, 
concern  with  author  background  characteristics  and  their  relation 
to  performance  was  seen  as  both  researchable  and  important  for 
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future  implementation.  With  respect  to  the  PLATO  device,  its 
various  cost  factors  and  overall  reliability  must  be  addressed. 

The  foregoing  considerations  gave  rise  to  Fig.  1 which  out- 
lines the  major  studies  to  be  completed  as  part  of  this  evalua- 
tion. Two  further  constraints  which  influenced  the  choice  of 
particular  studies  were  (a)  the  necessity  to  work  within  the 
guidelines  of  the  POMC  Implementation  Plan,  and  (b)  the  limited 
manpower  resources  of  both  the  PLATO  Development  Branch  and  AFHRL. 
SAMPLING  CONSIDERATIONS 

For  those  studies  involving  comparative  evaluation  of  stu- 
dents, a POMC  test  group  (mmbering  16)  will  be  selected  from 
personnel  entering  June  1975.  These  students  will  be  presented 
specific  problem  oriented  material  on  PLATO  IV,  rather  than 
receiving  the  "equivalent”  instructional  material  in  the  traditional 
classroom  setting.  These  materials  will  be  distributed  throughout 
the  entire  three  semesters  of  classwork  at  the  School  of  Health  Care 
Sciences  (SHCS) . The  remaining  48  students  from  the  June  1975  class 
will  serve  as  a control  group  and  will  not  be  exposed  to  the  prob- 
lem oriented  material  related  to  Respiratory  Disease  on  PLATO  IV. 

The  POMC  test  group  will  be  randomly  selected  from  the 
PA  students  entering  in  June  1975.  If  the  sample  chosen  by 
this  method  does  not  differ  significantly  from  the  control  group 
on  SAT  composite  scores,  the  sample  will  be  accepted  as  representa- 
tive. If  there  is  an  appreciable  difference  between  the  control 
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and  test  group  on  SAT  composite  scores,  a new  selection  of  groups 
will  be  necessary  since  it  has  been  shown  that  there  is  a high 
correlation  between  high  composite  SAT  and  success  in  the  PA  course. 
If  a second  selection  process  is  necessary,  the  entire  class 
will  be  rank  ordered  on  SAT  scores.  An  equal  number  of  students 
will  be  randomly  selected  from  the  high  and  low  strata.  Sample 
size  can  be  increased  to  32  by  following  the  same  procedures  for 
the  next  entering  class,  thus  increasing  the  power  of  this  design. 
STUDENTS 

1 . Academic  Knowledge  (Study  1) 

The  purpose  of  this  comparison  is  to  determine  whether  there 
are  differences  between  POMC  students  and  students  in  the  traditional 
curriculum  with  respect  to  academic  knowledge  of  respiratory  disease 
subject  matter.  If  the  POMC  succeeds  in  teaching  this  material  at 
least  as  well  as  currently,  then  academic  quality  will  have  been 
maintained. 

Since  slight  differences  in  emphasis  and  content  are  found  be- 
tween the  POMC  and  the  traditional  curriculinn,  academic  test  items 
from  currently  used  block  tests  could  be  unfair  to  POMC  students. 
Likewise,  the  POMC  generated  end-of -block  test  items  could  be  biased 
against  the  traditional  students.  To  resolve  this  difficulty,  a 
test  covering  respiratory  disease  will  be  constructed  consistird  of 
items  that  are  similar  to  those  found  on  the  American  College  of 
Medicine /Board  Examination.  These  items  will  come  from  the  Educa- 
tional Testing  Service  (ETS)  item  banks.  In  the  event  that  ETS 


cannot  supply  these  Items  or  more  Items  are  necessary,  experts 
In  respiratory  disease  from  outside  the  SHCS  (for  example,  Uni- 
versity of  Nebraska  PA  staff)  will  be  asked  to  contribute  test 
items.  These  procedures  will  insure  content  validity  since  these 
test  items  will  tap  knowledge  that  is  considered  important  by  in- 
dependent experts  in  respiratory  disease. 

This  initial  item  pool  will  be  further  refined  by  testing  a 
group  of  Physician's  Assistants  just  at  the  end  of  their  second 
year  of  training.  Item  statistics  will  be  computed  on  this  sample, 
and  items  for  the  final  form  of  the  test  will  be  chosen  in  accord- 
ance with  statistical  criteria  so  that  only  reliable  and  discrimi- 
nating items  of  appropriate  difficulty  are  included.  In  addition, 
this  procedure  will  provide  a normative  sample  against  which  scores 
of  the  test  and  control  groups  can  be  compared. 

The  final  form  of  this  test  will  be  administered  to  test  and 
control  groups  at  the  end  of  phase  I of  the  course,  thus  delayed 
retention  will  be  measured  for  both  groups. 

2 . Problem  Soi^ - Skill  (Study  2) 

The  skill  to  be  assessed  in  this  comparison  is  problem 
solving  or  diagnostic  skxll.  By  presenting  several  simulated 
patient  encovinters,  we  can  test  the  student's  ability  to  i.iuegrate 
his  knowledge  toward  successful  diagnosis  and  treatment  of  a patient. 
These  exercises  will  essentially  be  paper  and  pencil  problem  de- 
scriptions, but  the  student  will  be  able  to  request  information. 
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prescribe  treatments,  and  note  effects.  In  a limited  sense  he  will 
be  able  to  "interact"  with  the  simulation.  Scoring  criteria  would 
Include  correct  use  of  available  information  (lab  tests,  patient 
history,  physical  examinations,  special  tests,  etc),  correct  se- 
quence of  steps  toward  diagnosis,  accuracy  of  diagnosis,  appropriate 
indication  of  treatment,  time  to  completion. 

Two  or  three  Simulated  Patient  Enco\inters  will  be  presented  to 
both  groups  at  the  end  of  the  course.  It  is  planned  that  these 
simulatio.'.s  will  be  available  from  ETS  or  drawn  from  simulations 
which  are  already  in  existence.  It  is  felt  that  creation  of  sim- 
ulations by  the  PLATO  and  PA  staffs  should  be  avoided  to  eliminate 
bias , 

3.  Student  Attitude  Questionnaire  (Study  3) 

The  main  objective  of  this  study  is  to  discover  whether  POMC 
students  have  positive  (or  negative)  attitudes  toward  the  curriculum 
and/or  the  PLATO  system.  To  this  end,  a 26  item  version  of  the 
Brown  questionnaire,  used  by  Florida  State  University  in  a recent 
study,  has  been  selected.  This  instmment  has  demonstrated  reliabil- 
ity with  college  students  (r*.80),  and  Includes  many  specific  ques- 
tions which  will  permit  inferences  about  the  various  features  of 
PLATO  (e.g.  perceived  reliability  of  terminals,  depersonalization, 
etc)  . 

It  is  anticipated  that  the  attitude  questionnaire  will  be  given 
in  the  first  and  twelfth  month  of  the  course  in  order  to  detect 
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differences  as  experience  with  the  POMC  and  PLATO  increases. 
Analyses  of  the  relationship  between  attitude  and  performance 
on  the  final  academic  test  and  simulations  are  planned. 

4.  Lesson  and  Block  Tescs  (Study  4) 

Here  the  evaluation  will  be  descriptive  and  formative  in 
contrast  to  the  previous  study  which  was  comparative  and  summative. 
The  goal  is  to  show  that  POMC  students  are  meeting  learning  objec- 
tives as  they  progress  through  the  course.  Number  of  lesson  re- 
peats, number  and  duration  of  special  remediations,  eliminations 
or  washbacks  will  be  reported. 

AUTHORS 

1.  Performance  (Study  5) 

Author  performance  will  be  measured  in  three  ways:  (a) 

average  quality  of  lessons  produced  (as  judged  by  University  of 
Illinois  Personnel  and  student),  (b)  supervisory  ratings  of  per- 
formance, and  (c)  peer  ratings  of  performance.  Performance  dimen- 
sions to  be  rated  will  Include  programming  ability  (competence 
with  Tutor  language),  efficiency  in  lesson  design,  creative  use  of 
instructional  techniques,  novel  applications  of  PLATO,  etc. 

Questionnaire  instruments  will  be  designed  by  AFHRL  for  ad- 
ministration as  soon  as  possible,  so  that  time  comparisons  can  be 
made  to  show  the  effect,  if  any,  of  experience. 

2.  Attitudes/Opinions  (Study  6) 

Attitudes  of  authors  will  be  measured  primarily  by  a ques- 
tionnaire yet  to  be  developed.  Two  purposes  are  paramount  here-- 
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(a)  to  be  able  to  describe  specific  strengths  and  weaknesses  of 
the  PLATO  equipment,  and  (b)  to  Investigate  the  relationship  be- 
tween attitude  and  performance.  Dimensions  to  be  sappled  include: 
attitudes  toward  CAI  in  general,  feelings  about  individualized 
instruction  and  the  problem-oriented  approach,  human  factors  aspects 
of  terminal  design,  problems  in  lesson  design,  etc. 

Questionnaire  data  should  be  supplemented  by  interview  data  with 
interviews  to  be  conducted  by  AFHRL.  Also  open  ended  opportunities 
will  be  included  in  the  questionnaire  tormat. 

Two  measurements  of  attitudes  and  performance  will  be  made  at 
different  times.  Cross-lagged  correlation  analysis  may  yield  some 
Insight  into  whether  attitudes  cause  performance  or  vice  versa. 

3 . Characteristics  (Study  7) 

An  attempt  will  be  made  to  relate  author  personal  and  back- 
ground characteristics  to  performance  on  the  project.  To  this  end, 
data  will  be  collected  on  such  variables  as  level  and  type  of .edu- 
cation, age,  rank,  IQ,  attitudes,  volunteer /nonvolunteer  status, 
interests,  etc.  The  aim  here  is  to  predict  what  kinds  of  people 
make  successful  authors  for  possible  selection  purposes*. 

PLATO  TV 

1 . Cost  (Study  8) 

Since  the  PLATO  Development  Branch  is  a cdst  center,  aggre- 
gate operating  and  development  cost  figures  can  be  readily  obtained. 
Management  analysis  personnel  will  be  contacted  for  consultation 
purposes.  Capital  investment  figures  should  be  available  from  A*IPA. 
A projection  of  expansion  costs  under  several  scenarios  should  be  a 
part  of  the  final  report. 
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2.  Reliability  (Study  9) 

Logs  will  be  maintained  to  show  number  of  terminals  In 
connlaalon  per  day,  percent  of  down-time  per  terminal,  cause  of 
failure.  Time  phased  analyses  can  be  made  to  detect  trends. 
STOflURY 

Figure  2 gives  an  overview  snd  summary  of  the  plan  with  re- 
sponsible agencies  and  deadlines  specified. 
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date  given  is  deadline,  interpreted  as  Not  Later  Than 

indicated  in  parenthesis 

completed 


APPENDIX  F:  TESTING/GRADING  PROCEDURES 


DEPARTMENT  OF  THE  AIR  FORCE 
School  of  Health  Care  Sciences,  USAF,  (ATC) 
Sheppard  Air  Force  Base,  Texas  76311 


MSDM  11  Sep  75 

Testing/Grading  Procedures  for  Students  Participating  in  the  PLATO  Project 
MSDM(LColSeager)  MSDM  (Cdr  Peterson  MSOR 

1.  PLATO  material  has  been  developed  whidi  is  designed  to  meet  specific  objectives  taken  from  the  Flan 
of  Instruction  for  Courw  3ALR91730  (Physician  Assistant).  When  lectures  are  given  whidi  address  these 
same  objectives,  PLATO  students  are  to  ^ excused  and  report  to  the  PLATO  classroom.  Because  the 
PLATO  project  involves  an  experiment  in  curriculum  design  as  well  as  the  application  of  computer-assisted 
instruction,  the  sequencing  of  material  presented  via  PLATO  will  not  conespond  to  sequendng  in  the 
regular  curriculum.  Hence,  PLATO  students  may  not  have  had  the  material  that  corresponds  to  lecture 
material  from  whidi  they  were  excused  vdien  scheduled  testing  takes  {dace  in  the  regular  curriculum. 
Therefore,  HQ  ATC  has  granted  a waiver  of  the  provisions  of  ATCR  52-3  for  students  who  are  part  of  the 
PLATO  “test"  group  so  that  normal  testing  procedures  can  be  modified  to  accommodate  them. 

2.  Ihe  follow  ng  ^ocedures  apply  for  testing  these  students: 

a.  Prior  to  L'l.ninisteiing  an  examination  or  quiz  ^ich  covers  objectives  for  whidi  PLATO  materials 
are  Involved,  the  PA  faculty  member  planning  to  administer  the  measurement  instrument  w31  contact  his 
PLATO  staff  counterpart.  Jointly,  the  PLATO  staff  member  and  PA  faculty  member  will  determine  which 
questions  are  to  be  exduded  hrom  the  examination  for  PLATO  students  and  MSORM  will  be  advised. 

b.  At  the  time  of  the  examination,  PLATO  students  will  be  directed  not  to  answer  the  questions 
identified  by  the  procedure  above.  Ihe  instructor  will  write  the  numbers  of  the  questions  to  be  exduded  on 
the  Uackboard  and  have  students  in  the  PLATO  group  drcle  those  numbers  on  their  answer  sheets. 

c.  At  the  time  the  examination  is  graded,  % scores  will  be  calculated  for  PLATO  students  on  the  basis 
of  the  number  of  questions  answered  correctly  out  of  the  number  of  questions  not  exduded.  For  example, 
if  five  questions  were  exduded  on  a test  whl^  normally  contains  50  questions,  the  PLATO  student  score 
would  the  number  answered  correctly  divided  by  45. 

3.  The  PLATO  staff  is  responsible  for  testing  objectives  covered  by  PLATO  material.  Formal  tests  will  be 
administered  u often  as  three  times  per  trimester  and  the  results  of  these  tests  will  be  used  to  adjust  both 
raw  scores  and  percentage  scores  of  tests  from  which  questions  were  exduded.  Ihe  following  procedure  will 
be  used  in  adjusting  these  scores. 

a.  On  the  basis  of  the  number  of  questions  not  excluded  in  comparison  with  the  number  of  questions 
normally  contained  in  the  test,  a weighting  factor  will  be  determined  for  each  test  in  the  re^nilar  curriculum 
that  was  affected  by  the  procedures  outlined  in  para  2 above.  For  example,  on  a test  which  normally 
contains  50  questions  from  whidi  eight  questions  were  excluded,  the  weighting  factor  would  be  .84  and  the 
weighting  factor  for  the  PLATO  test  woidd  be  .1 6 (so  that  the  sum  of  the  weighting  factors  equal  1 .00). 

b.  At  the  completion  of  a PLATO  testing  cyde,  all  affected  test  scores  (percentage)  in  the  regular 
curriculum  which  h^  not  been  adjusted  previoudy  will  be  adjusted  according  to  the  following  formula: 

Test  weighting  Test  PLATO  weighting  PLATO  Adjusted 

factor  X Score  * factor  X Score  « Test  Score 

Raw  scores  may  be  adjusted  by  the  formula: 

Ai^usted  Total  Questions  on  Adjusted 

Test  Score  X Test  before  exclusions  •=  Raw  Score 

Attachment  1 of  this  letter  contains  a completed  example  of  the  procedure. 
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4.  The  above  procedures  become  effective  Immediately  and  wll  remain  in  effect  for  the  duration  of  the 
PLATO  project. 


NICHOLAS  C.  NICHOLAS,  Col.  USAF,  BSC 
Chief,  Department  of  Medicine 


1 Atch 

Example  of  Procedures 


EXAMPLE  OF  TESTING/GRADING  PROCEDURES  FOR  PLATO  STUDENTS 


Test  in  reginlar  cuniculum:  SO  Quettlont 

PA  — PLATO  staff  memben  exclude:  8 Questions 

PLATO  students  responsible  for:  42  Questions 

PLATO  student  who  answcn  correctly: 

# of  Questions  % Score 

42  100.0 

40  95.2 

30  71.5 

Wei^ting  Facton  determined  by: 

# of  Questions  Total  Questions  Wei^ting  Factor 

not  excluded  on  Test  * for  test 

42  SO  - .84 

1.00  - Weij^ting  factor  for  test  “ PLATO  test  wei^ting 

factor 


1.00 

- .84  - 

.16 

Adjusted  Grade: 

Weighting  Factor 

Test 

Weic^ting  Factor 

PLATO  test 

for  test 

X 

Score  + 

for  PLATO  test 

X Score 

« Adjusted  Score 

If  a student  received  a score  of  90%  on  the  PLATO  test,  his  score  would  be  adjusted  as  fdlows: 

Wei^ting 

Test 

Weighting 

PLATO 

A^usted 

Factor 

Score 

Factor 

Score 

S^re 

.84 

X 

100.0  + 

.16 

X 

90  “ 

98.4 

.84 

X 

95.2  + 

.16 

X 

90 

94.4 

.84 

X 

71.5  + 

.16 

X 

90 

74.5 

Raw  scores  can  then  be  adjusted  as  follows: 

A^usted 

Total 

Adjusted 

Scoie/100 

X 

Questions 

m 

Raw  Score 

Unadjusted 

A(^usted 

Total 

Adjusted 

Raw  Score 

Score/100 

Questions 

Raw  Score 

42 

.984 

X 

50 

■ 

49 

40 

.944 

X 

50 

■ 

47 

30 

.745 

X 

50 

■ 

37 
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APPENDIX  G:  ATTITUDE  SURVEY 


Instnictions:  Rease  answer  each  of  the  following  questions  as  objectively  as  possible.  There  are  no  wrong 
answen  so  give  only  your  own  opinion,  not  what  you  think  someone  would  like  to  hear. 

Key:  SD-Strongjy  Disagree  D-Disagree  U-Undedded  A-Agree  SA*Strongly  Agree 

For  each  of  the  following  questions,  circle  the  desired  response. 

1.  While  taking  Computer- Assisted  Instruction  1 fdt  challenged  to  do  my  best  work. 

SD  D U A SA 

2.  I was  coioemed  that  I mi^t  not  be  understanding  the  material. 

SD  D U A SA 

3.  I was  not  concerned  when  I missed  a question  because  no  one  was  watching  me  anyway. 

SD  D U A SA 

4.  While  taking  Computer-Assisted  Instruction  I fdt  isolated. 

SD  D U A SA 

5.  I felt  uncertain  as  to  my  performance  in  the  programmed  course  rdative  to  the  performance  Of 
odiers. 

SD  D U A SA 

6.  I found  myself  just  trying  to  get  through  the  material  rather  than  trying  to  learn. 

SD  D U A SA 

7.  1 knew  whether  my  answer  was  correct  or  not  before  I was  told. 

SD  D U A SA 

8.  I guessed  at  the  answen  to  questions. 

SD  D U A SA 

9.  In  a situation  where  I am  trying  to  learn  something,  it  is  important  to  me  to  know  where  I stand 
Rlative  to  othen. 

SD  D U A SA 

10.  As  a result  of  having  studied  some  material  by  Computer-Assisted  Instruction,  I am  interested  in 
trying  to  find  out  more  about  the  subject  matter. 

SD  D U A SA 

11.  I was  more  involved  in  running  the  machine  than  in  undentanding  the  material. 

SD  D U A SA 

12.  I felt  I could  work  at  my  own  pace  with  Computer-Assisted  Instruction. 

SD  D U A SA 

13.  Computer-Assisted  Instruction  makes  the  learning  too  mechanical. 

SD  D U A SA 

14.  I felt  as  if  I had  a private  tutor  while  on  Computer-Assisted  Instruction. 

SD  D U A SA 

15.  I was  aware  of  efforts  to  suit  the  material  specifically  to  me. 

SD  D U A SA 
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16.  I found  it  difficult  to  concentrate  on  the  coune  material  because  of  the  hardware. 

SD  D U A SA 

17.  Questions  were  asked  which  I felt  were  not  relevant  to  die  material  presented. 

SD  D U A SA 

18.  Computer-Assisted  Instruction  is  an  eflident  use  of  the  student’s  time. 

SD  D U A SA 

19.  While  on  Computer-Assisted  Instruction  I encountered  mechanical  malfunctions. 

SD  D U A SA 

20.  Computer-Assisted  Irutruction  made  it  possible  for  me  to  learn  quickly. 

SD  D U A SA 

21 . I felt  frustrated  by  the  Computer- Assisted  histruction  situation. 

SD  D U A SA 

22.  Ihe  Computer-Assisted  Instruction  Approadi  is  inflexible. 

SD  D U A SA 

23.  Even  otherwise  interesting  material  would  be  boring  when  presented  by  Computer-Assisted 
Instruction. 

SD  D U A SA 

24.  In  view  of  the  effort  I put  into  it,  I was  satisfied  with  what  I learned  while  taking  Computer-Assisted 
Instruction. 

SD  D U A SA 

25.  In  view  of  the  amount  I learned,  I would  say  Computer-Assisted  Instruction  is  superior  to  traditional 

instruction. 

SD  D U A SA 

26.  With  a course  such  u I took  by  Computer-Assisted  Instruction  I would  prefer  Computer-Assisted 
Instruction  to  traditional  instruction. 

SD  D U A SA 

27.  I am  not  in  favor  of  Computer-Assisted  histruction  because  it  is  just  another  step  toward 
depersonalized  irutruction. 

SD  D U A SA 

28.  Computer-Assisted  Instruction  is  too  fast. 

SD  D U A SA 

29.  Typing  experience  is  necessary  in  order  to  perform  easily  on  Computer-Assisted  Instruction. 

SD  D U A SA 

30.  Computer-Assisted  Instruction  is  boring. 

SD  D U A SA 


APPENDIX  H:  PROFESSIONAL  STAFF 
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